Anda di halaman 1dari 18

EVALUATION

MODELS
VIEWPOINTS ON
EDUCATIONAL
AND HUMAN
SERVICES
EVALUATION
SECOND EDITION
edited by:
Daniel L. Stufflebeam
George F. Madaus
Thomas Kellaghan
Kluwer Academic Publishers

Boston I Dordrecht I London


Distributors for Norrh, Centr3l and South Arn€rica:
Kluwer Academic Publishers
101 Philjp Drive
Assinippi Perk
NoNe[, Mas.chusetts 02061 USA
Telephone (781) 871 6600
Fax (781) 681-9045
E-Mail <kluwe(4)wkap.com>
Distributors fot a[ other countries:
Kluw€r Academc PublDhe6 GrouP
Distiibution CcDtre
Post Ofice Box 322
3300 AH Dordrecht. THE NETHERLANDS
Telephone 31 78 6392 392
Fax 31 78 6546 474
E Mall <services@whp.nl>

\a
Et"*-"i. services <http://wwlwkap nl>
f\

Library of Congress Cataloging-in-Publi'ation Data

Evaluarlon models: viewpoinrs on educational lnd human serrices evaluation / edited by


Daniel L. Stulllebeam, ieorge F Madaus,Thonus Kellaghan 2'J ed
p.cn1.-(Evaluation in educatioD and bunun senice$
lncludes bibliosraphical relercnces and nrdex
ISBN 0-7923-7Uti4 9
1. Educatlonal sunq's 2 School management and organization-Evalu:rtid 3 Curriculur
evrluation. L Stuflleberm. Danicl L. lI Madaus. George F lII Kellaghan' Thonas IV Ser:s

L82823 . E87 200t)


179.1'58-dc21
oo,0348ir

Copyright O 2000 by Kluwer Acadcmic Publnhcrs


All rights reserved. No part ol this publication mev be reprcduced, sDred in a retievll systeD or
transiritted in :rny forn or by any n'.ch""i."t. photuopvins, recordins' or otherwisc wirhor
'r.a.., Assitu;a
the prior wrltten' penrxsion of the publisher, Kluwer Academic Pubhhers, 101 Philip Drivc'
P!rk, Norwc , Massechusetts 02061

Pikted or d.id-Jke fap\.


Printed in the Unrred Statcs ofAnerica

T'tt llehlkhet one$ seouflB on thi' book Jot roune Be dnd bttlk pnfthases'
Fot Iurthet infonation, eflait to <'flichael.witians@nhap'con>'
'end
::-: -1 :: .._ r.\acnlivcly in the
: :: i..t :.!. i.,r crantplc, ttred
.. ' - j\r,,\\ lro\\'extcrr_
::. 5. OUTCOME EVALUATION

:-- . :: ::: ic\eriil chanses in


_:: : .:: :r',c problbly rnade THOMAS KELLAGHAN and GEORGE F- MADAUS
::-::_ : : :i:nl sidely usetl ln
::a:-: .: :i :ror oftcn used to
' .J!tL.1 !,n.11 sltLl I tto
. :i::..::: :: Ihus. therc is eval-
- :.:.::::j the conception of
::.=.:. :-.::toni are rpPrcPriatc
- - i.: ::i:rnq ou! of CurriCu
::: :.i-: :f (-1, a hrger plr.t of
: :: ::. E:qht-Yeat Strrdr., but
::: : Ii.u. rirerc is the eval
:-.:: :: :h. c.,'elrrrdon oLthe
.:: -t: .t:t;ti.,tl1\' plescnted in
::: :. :.tlultion in rhc con I
: .:::... .irlier iinpro\renrcnts The flrst cdltiol ol Ernhntiott ,Vodrls dld not corll:rin :r chapter on outconre eval-
/
:: ,: ::tai oi il progrtnl, :ts Llalioi. Whv is there onc rr this eclltjon? Aitcr :rjl, rhe jdcl of rDeasuring otrtcorrrcs
.. ..: -l:.i rcsulrs :rrc beine rs not nerv and. es rvc shall see. outcome efiluatior) c:rn hardly be rcgardecl as e
- .:::::in rhc long tcrlt unitary approirch. qiven thc varic-tv of practrccs cncomp:lssed b_v tlic term. Nor cen
-:i:.: :. f!rr)re of n,hich lre ir rcalll bc considered a rrrolr1. lf bv niorlel u-e rne:ln r nror.c or less elaborltc rcp-
resenl:ltion of rhe srructtuc and rcllrioDships of a ra gc of phenotlena. Sotrre rvoulcl
'.::: .:: :.luirtional progranr
- say ir ls not cver evrluaaion. Horvevcr, llrere is srill good relsorr for includlng il
:: : : ::::ri:rl1i|g ancl appr-eis_ rhis volume a descriptlon oi:rctiviries thet can be broadl,v crteqorized as outconle
--- :t:-: iit le.rrntng itt addi er':lluedon, sirrcc- tltcy Dow .rccollnt lor I considcrable amount of progr.un nroni
: : - ::.t:tcr. lhC necd to toring ectivities throuqhout thc vuorlcl, iD sonrc cescs displacitig ntore tr:rditio al
-.,: ::: rrltic. the lcarncr! approaches to evrluetlon and rese;rrch. both in counrr.lcs rvrth long estebllslrcd tra
,:-: ..: i::irl.lrds thc teacher- ditions in these disciplincs rnd oncs rvhere fbrnral cvaluation rtivities ar.c orlv being
--. : :]lo,e cnvirolltlentel dcvcloped. C)utcome evelrratiorr hes rcccl,,c',:1 the backrng and lirarrcitl support of
:-- - ::::.ild since 1929. governmenls ils well as of iitern:lrional orqanizetions, such es the Etrropcan UnioI.
--: r:: ::.i\ Lrnds of behav_ the Orqanisltiorr for Lconomic Clo oper:rrion lr d.l )cvcloprncnt, UNES(l(). and t|e
--,::. :::.] intcrcsts itt various Wor'ld Benk.
i r ,:::_.r'-\ oi thc purposes, In rhis chaprcr...vc shirll iirsr dcscrrbc rh. .h".,,.i.r,rtr.r ol oLrl!:!,nre evahtation.
: -..:..:..rr \\ill continue ro
-
Follor'lng thlt. t'e sh:rl1 outlinL' rcasons lor irs glorvrh .rrd ic1\'dltaqes attributed to
its use. We shall then identih. a number of trildjrjo ! ind clevelopnlerlts ro which
current prectice in oLltcolDe e\,:rluatjorl is indebted, follor""-ed by exrnlples of
oLlrconrc- c-\raLu:rtioD Nt strte, rrrtionrl. and irtrcr etiotrtl lcvcls. Alter that. .,1,c shall
consjder rpproechcs used irr orrtcontc c-v.rluation. In our conclucli q rcrrNrks, rve

t/l,lii,,r lr,ri,,r. ll/ rir/rrj r.,r ri


98 Il. Qu.(i.!\ri\l er]lrd\ (lrrcnr.(L lliluroot NloJ.l'

sllrll outlir)c i1 ltulrlber oflssues r'.risccL bv ourco tc cr':tlullion. ,rrd corrsrrlcr hoN it
lrrs euong rrldition:rl lpploachcs. l\,lost oi our
jllusrrativc nuterilLl Nill ,,rrrre liorrr
rlic flelcl of cchtc.ttiotr. nllerc otttcotlle cvalrLltillrl pnrtr:lblv has h]d lr\ grtiltcil
inrprct. Llut such er.llu:rtioll is b! n" "ttt"t '
orrtrrrerl to educltlon

\X'HAT IS OUTCOME EVALUATION?


A nunrbcr oi fcttttres ot otlt.ollle cvlrlultlon c.rrl be icl''lrrificd l-irttl)l rt r\ 'r rcrrl1
rll:rt rs lf.pliecl lo ilcai\ltic( tlleL ere tlcsiqlrccl prirrrarilv to 1l1c'rsrLlc lhe (oircn prc-
,urlr.,l) .f1.,l,r, or-rcsults ofploqr.r]lrs. rlther th:tn lhci inPllts or proccs\es' Sccolld'
!i CL' lt{I.c th.ul tlt!,.LsLlt.ctlt(ttt is r.etIllrccl i1_:rn rcrivit,t rs to bc reqlr(lc,l .r'i cr',r]ur
ti\1-. ijudglrreDl ls tr:r $hcrc'lr product lies rvirh resp.-ct to I strlirl'rrd is oficn rlucle'
(
Thus-,,,utcorrrcs nr:rv be Iclltccl to :l t;tlgcr. st:rndar.l oisclvice' or echievcrrrinr )tlcrt
rhc icle.t of crc.'llclrcc" i\ uscd or implicd Thc rridcsprc:rrl usc of lhe rrcblrlous
tcrrlr .rvorll cl:rss st;rnd.rr.,js" bv rltosc l rhc st.rnd:rrds-besed tefirrrrr rr'orcrrr.rrr itl i.:
the U.S. is npicel oi !h1s raaclll oll crccllcnce Sonlctirncs rhe juclr:nlenr of rnelit
or wortlr is inrplicrt r'.Lthcr thrrr crplicrt Arr inrpLcit 'Jucl$rr''nr is i uolvcd u'hen
inior rrr:rtiorr on ollt.olllcs (c.g.. tlre ,rr,,ttr :r-Lier.cnlcnt lcvcl oisrudents rrl r school)
ls norrrrltive (c.g.. inclic:rtin-u r"llerc lt school slillrd\ rclxlivc () otlrcr s'hu"lsJ rnJ it
is lcir to clrenrs .rrl.l tlrc ptrblic to nlekc tllc cveltl:ltlve jndgnlent :ltlt1' pcr-h:rps ttr
rrkc rcrion.
Thircl. rhc r':lrlgc- oioLltc()lrles thrt hirl'c bccll nse-d in outcortle c\'lllr:rtr\)n i\ totr-
siclertble. wrthin tlrc ficirl of eclucltion. ecrclc' tic rchic\.eurent is lhc ottlcorrtt' llrost
hequeutl-v :tsscssed. erttl I rerien oi perfbrnlrn'::c :rn':1 portfolio nlories heve bccn
c.rplovcrl $'itb rrrirecl success Mosl \tiltes rlow erl4rlor' \\'rililic srnlple\ :llld thesc
1,".c L.'.,, rnore succerslitl. C)tbcr pcr_iirrnl:r;lcc encl portfilio assesslllcnts hor'cver'
heve provecl ro bc incilicicrrr. cosdr'. lrrd ,t1116lj3b1e Kentuckt' hlcl to drop its pcr'
lbr nr:rncc lsscssrlrt'rl1. \rhile \icrnlonl hld to rethlllk lls rclilrlce on pollfi)lios
(Kortcz. 1r)91; Korctz, B:lrlon. Milchcll, & Stechcr. 19()6)- Other orttcotrres hl'c elso
bcen co sidercd rc1:tttng to builciing. erltLc:rtiorratl nlirler iirls. tclching. ,lttltudes to
.r^^1. 1.'r .. r'.r\.r ''r'. 1''t, ll.r' Irr_\l l \l(.r' ' rrllJrr rrl(''
'rr 'r "
rbscrrlceisrrr..rnd sludenls'losl \cLlool dc\dnatloll\) Forrrth. tllc eftects or results th:rt
.trc thc tocLrs of outcomc cv;riltrri,:rtl lluv bc obscrvcd :lt v;uvinq l"rntr ir 'L

progrrnr dulirrg its liti'. et it conlplclion. ol lxtel itl lirlrc to rsscss loll1l tu rrl1 eft((t\'
N'lost iicqucrrrlr,. thc tircus is oll olltcolllcs ll rhc aonlplclloll of I progr':tlll'
Fiith. lt is r)or usuJi itt oLltcolrlc c\'llLriliclrl to \cck t() dcscribc or spccily rvhet
is:rcrtL:rll,v hlppenitr* |t e Pro{rrrrt. though rhc kinc] of infol rnetilrn obt:rinerl rvill
obviolrsl-r'. irr gcuer'.rl telrlls ra lclsl. hc cholcll lo l-cllect progr:lrtl ;lctivilics- ln ll:irl-v
crrcumsluces ln wlrich outconlc c\,ilLltrioD ts uscd- I descriptioll of PloLr.lrll Jrtiv
itrcs {,ould bc verv tliflicult, if rt 111 possibk'. Thir is becluse nlalv proqr'IllN arc
extrcmell conrplcr rncl crll onlv bc corlsiderccl froqranls ir) the bro:rdest sr:rlse of
rhe rvor cl (:.g.. elcnrelrt.llv ccltlcation) Sttch proelenrs lle pe raps lllolc ic'ur!lclv
dcscribctl es complc\es of progrlrrlls, uhich lrc irllplcnrcrttcd in a vlrictv oi rvr,vs'
nrd hrl rvhich lhL' lcrlll \vslr'lll lniqht bc nlore afpropli:lte
Sirth. $'hile ()Lrtc()rrlc cvlluiltiolls lllirv e\chc$' dcscriptiolls of ptogr'rnl 'r'rr'ltitt,
--

:. OL,r.orr. t:lilLritLoi 99

ilctors or to prestllllcd ]ele-


r..: : :: :::.r .on\idcr how i! eflort\ Drrv be made to relilte otltcortlcs !o colite\tllill
the c\tL'nr to thcY
\,allt antece(lent r,.rrjlb1cs. Evalultron\ \'rr\
_ :: - :: .::::., \\ill conlc frol11 gLcJll\ rrr
,rvhith
t":n"tlt:i
shrll rct'r ru Jrrrl)trL'rl
:: - .r.i! ir.rd its greatest ,,r.urPa',.-a. Utt-, artd' letcr irr thc prPo' rrc thc nl;rin Pltrpo.se-l-'',:,'{-l
ll"Ji l the issrte W|cn sucLr rc'hLrrques rr': u'e'L
"aa,-* gross ancl lrct c'ilecls of prograltr actrltt1
attra*rt-f. itl oulcoltle data bctlrt'cn the rs it lrequentl-v is'
iLrlirnnrrtiorr is.to bc usccl
,;'t t|l;;.;,t, to clo this il outcomc jn or tor ilccoLrrll
iu ,l,a ,tt"rt,,gantan, of lesources conlnrl' ful
qualrry assurlllcc-
r:::: :: :-:. Frrrd\: it is il terrt)
to tlrc' perforrnrrce of insti
' : :: i:i,:ri chc (often pre ;ri;,; o;rP;", (c s . !o rctosnjze 'Lnc1 att'rch '"'ttti"n'
'-Ar"ra*ll\ \\,rrh re\p!,n\llrilit,v for !he rnrplcmcrllatioll of ir plogralll)-
::: :::.1. :: _:TOCeSSCS.
SeCOnCl. ,",r"ar'",
cvrlurllolt Irrav t)e oncc olf or rrw inr'olvc nlonitoring (l e '
- -_. :rg:lraied as evalue- Finalh', outconlc'
- ' ' ,:,::;.rr':l is often nrade. comperisons oi outcotltes o\'cr tinlc)
When integrated into a pcrlbrltlance
' :: :. , : :.hiclcnrent. C)fic'n svstcnr, it is Lkcl,v to bc tllc htter' sincc
it bls to frt into erl otrgotnu
","r,lr.at.t,,
:::: :r: -:ii of dre nc'bulous :rco\'lt),
:: :-. : :::irrrrr nrovemcnt in lN OUTCOME EVAIUATTON
REASONS FOR GROWTH
::: ,. .:: -tLr.lgrrrcrrt oi ncrit
qro*'th ir ottlcotlc evelu'rrioi Frrrr' tron
: _-:i-:.::rt ir involvccl rvherr A nunrber of reasoDs Can be iclerltiiicd lor Survc'"" corD
.: :.: -: r:Lr.l.'nts in r school) poinr of vie$, rhc 19(,r, E,.pr:rl Edrrt rti"rr.rl L)PPornLrrrr!,
""'irt,,.,:."t ofcdntlo'rn'Ll policvmirkers
-r :- : ., orhir- schools) and it rronlv clllerl the Colt'ulln ttpo" t"t,\tti th' JtLclrtl')rl
',"r'- '*"tttl.1.i"titt"'' t'rtqu"r c'ducrtjonrl .ppo't..itv in lertrrs trischool rcv^Lrccs
: : -.-:-:rr.rlr rnd. perhaps, to l'-v tests (oolerrrart et rl ' 1c)66)'
:.'].ii...1';;;:;,;;-:i".'t'o"^l oLrlcorrres as ,'''""""1 :lpproaches
r' :. :r!: c'\irlurdon ls con *",.,,' is lhe !'erceivecl Po!'r lLttnd oi |r.Lclitl(lnrL,crJlLl.llr,n)
nrr tlrt I Ir- rrrrr' Lcr
:: : :: : rr the outconle nlost in p-tlalr.i ,lrr" ror l 'r pur'' rr-'lL rr' ' rkrr'e 'l '"'r ' 'l
^',"..,'o
::: _:'-:.r rnodes have beerl
"r!*oi. o;"**,', rh.rt rr.*' b". der;::t'::,]il;i::l,t:::l":,'i,:,:.;::1,'4"',,'i:ili
r :r::::: i.lnlples and drese appliclblc solutions clid not secur t llcl
:: :: : :-ii!'\illlr-ilts. however, & I)eltc- 1996), *'hile "ttttt-u tt'"ltt"tlt"t"t't"' 1tt'ctiuttl to bc costly" slou"
: l^
..rr pl \. rur L, r', r 'ttt;t 11 "r' ri r' ^l'l\'ul
rc\'
:,:: : -,: . r.r.l ro clrop its Per
.:. :: iir1.c on portiolios Athirdreasotltorthcgtolvrhirrt'uittu'ttt""alttatio"istheder'cloPrrlerlrofl
slgnllecl bv x lise in "lu:lll:lge
- :::r: olllconles htVC alSO (:orD(tilist apploach to govelnnlclll adnrirl istrltjor '
i'Ju'n..d br rdr '' h.r' th' bu'rrrc"' rvortl involv-
::::-: -. ...rihinq. irllitui:les to ;i:iilil;i';';;"'f i''r'''"'ilvp1t""'ng' rrs' of pcrlornr'rcc rrrdrcators' I tocus
: : r ::-:.ia]]r retenlion rates, ;;;:;;";;i. ,iil .,p""tio''"i 'l'" rcce^Lnr LL'rlrn sl stens b:rst-rl on
r -:: .:.: .:ricts or results tltar :.;i";:i;;;i"t",';ults :r grorvtlr u' 'rnJ
'n'-'"r"" (rrrLicPt {'t !l"1:l' ;rs corlsunrL'r
:.: - :.:\inq poirrls i[ a i"*Ut i.* .'t,).*U"tt'tttt'' t"i"t"t] prr) 'nrd the ':' rlrrrrr'r rdrr'r"'rr
rlr^
lr' lrr tlri' rrr' r"'r' rl rr' rl(r'rrr .l'1' "r r

-- :i:ii long terlll cflects. n ' . l'r'u"'u.. '(;;;31


l,'' '
r'ecf is bcr'g rlispleced' .rd-its succ"ssor is 'urrtt-
:a:: r: r: I Fr-Ogr.llll. ilrilfi^ "hs''
anil-vsir" (p 35'1) The nlrrlasellenl
rr : ri.-:rirl' or specitv lvhit .*.,,Ut"rr, tto, O,ct'c."""t' trrlrration '1Dd
: : ::: :::.:.lr()r) obtained rvi1l ccrl]sultltrtlscrpectecltoL,..or"toproviclethequick,narrorr'-fbcusedanahsislillri
i::: .. j:.:: :1.fl\'ities. In malry '" neecled.
is
cvliuatio.
.
is dle rncreasrr' rg i.fltrence of
: r:. -: :::. :r ara ProerrlD ilctiv- titl n reaso. fbr grorvth in oLtlcorrle
audit comntunirt' in non-f_Ln'rncial areas
of public adlninistr:r-
:. :- .,:.: :il.rr)\' progranls arc the^ lccot1llli11!l atrrl fbr nrolev audits"'
. ::: :::. brorclest sense of tion.'Ihe inllucnce is rt'llccteri itt'jtt'utp'"1t""'i"":rudits"'"r'llue
" oi cor'rlltr1es loLliy
''pelti)rmanac .,uelir' ' ''nd ttt'it'tt"''tl luclits In a variety
,:: :::1.:f! rrrore ;rccur;rlelY
r:r .r rlricw* of u'a-ts, 1, g..rl'e L r.rrrJ 5r"t'
rti I""''rr'cJ fu f't' rr Nru 'zr'rl 'r'd 5utJ'r" 'uJiL' "lpcr-
r'r' rr( r..r!'1 !rr 'l* crr"rr'o *I'iclr
1.,rrrrar,.c'rr,l', 'r(rrrr((1 ";' ''"i"'''""' 1 999)'
,;:;",1,.-;; ;;..*','"ts ire r,,ecrirrg inclicetor t:rrgcts (Davics'
l(,rr ll t, ., \,,r..r ,.. |. \,.

Fililrl-ri gr-o$th i' ourco'rc' c'elrL.rtio' rcflects iucrcrsi'g Lr\c or':rsscsl'e'r irs a
policv rool. Irr rhe fic1cl of crlrrcrrion. rhis rnvolrcs a shiti liorn ahL- usc ol- lsles\
rrenr l^ti)r':rtron lir' locllizeri rrst.nctior:rl clccisio' .raki'g to cenrrllizcd high
strkcs poliq, niatins rrd lccorLnrrbility nroli.orirls (Mldaus & Raczt,k, 1996).
Sirth. rhL: enrn'th ot oLrtt.oure cvelulriou orrcs ntLrch to it r-eorglrrization of the
fribiia \ervi.c irr severll cou'ores. tcs'lring lrr the usc of relrtl'elv :tLltonoltloL^
service ptovitlers (e.g., Nitrio itl Hc:rlth iicrr,icc fiusts l rl qt.;tnt-ntrirtrel cd schools
ir llritlrrr). With rlccentr:Lliz.ttion of pr.og.rr' :lLlthorit\: .1ld the conscquenr loss .f
dilcct contlol ovcr thc irDplcntcltrrjorr of progrtits. the ncc(l llosc fbr ncu. coD
tractull alranqenrenl\ \\'ith rclvicc prolidcrs :rnd fbr. regulidon :rnd cornpliencc
nrolriror-ing. "Qrrilitr"' and "st:rndtrcls" itre thc' thcilre rcrnts. .lrrci c|illuittroD .rrrrltLc
rttcnts erc dcsiqned to chcck tltilt organizetiols :rre dcllvcring flc\iLrle. cost cflc.trve
ser|iccs to cirizcrr users (l)o1li.t. l9cl3).
Frnrllr: I sit.ation i. *fiich ero*th i. ilen*n,.l ibr p'blic ser'iccs rrcl socirl
progrrrr lirnding (c.g., eclLrcltion, he:rlrh c:r1e. s6r:ill sc,cprirr,) rs qrog,irrq nrorc rapid1.,
thi'rr-sou'ces a;lr) be fbu.d lirr crplnsior L'.rr1s to rhe |ced lbr grc:Ltcr.etiicicrc,,,,
rvhich in tlnli c:rlls fbr sclecttriry l'tlecidirre \ ttt pt.ogr:ltns:tre to be co.till.ecl
iud rrh;r! lc\\' acrivities arc- to bc hrrncheci (l3ll1ock. 1999; l)ur.atr, Monrier. &
Snrith, 1995r lrollin. 1993).

THE VALUE OF OUTCOME EVALUATION


Sc'c'rrl rd'errrlgcs hlr.c bcc'' itttribLttcd ro thc, .se of o.tcontc e'illu:l.olt. ()tlc ls
based on business exp('rience, trltctc $cll-articuhtcd qolls:rrc lssocilrcd *.ith otglr
niz:rrio.ll e0ictiv.r css. Thc situ;rrion ir sciroois. '.rhich rre rororious irr lecki.q such
goirls. stands ir sf(rnq cotrtrrst to this. lt is ireueLj rhat ii_ schools rvere
to spc,cih
o.tc.'res rchri'g ro rao:rls. this *o'ld iclcntih r,,,hlt is i'rporta,t. rnd *.otr1cl
help t,rcrrs teiahcls 1rd st.clenrs or cs!c'rirr cu.icLrru'r conter)r (sce salr'riLrr.
McKnilrht. & R:rizen. 1996). k ir llso rhe posirion oiadvocrtcs of our(:orre evlllt
:ltion thirl the s|ccltlclrion of oL[cont.-s ir likcll to l]:1\,c i grettcr lInpect NheD
:rli4rcd *'ith ;rppropr-i:rte is\csslltcrll. Tliis oricDtlti.n ro*.et rl specifj.ing oLrtcol'r,\ of
sclrooling is tt drc hcirrl ofthc st:rndlr.ds b;rscd r.eftrr' .lo'cr1re.!.VlrioLrs strtes hevc
,:le'elopcd c.rriculurrr lil'rc*.rks tiril rliln.l:rte, hrst. lc:rde.ric l.:u..lrrq st:'dlr-ds
b_v !lr'.td!-:uld strLrjcct:rrcl, lr'd scc.nt1, its\ess'tcr)ts ao ltlcrsLlrc,:rchic'cntenr rchtecl
to thesc firrrrc\rnks. For crarrrplc. the N4asslchusetts Clontprchotsive Asscssltlent
Srstcrrr (M(iAS). I Iew rs\cs\l11c.t progrJltt li)r prLirlic schools. , meesurcs rhe pcr_
firrllunce oi stuclcnrs. schools..rrrcl distticrs on thc lcrdctDir: IclrDing stilrtlltrds corr
t:ri.cdi. drc l\lessrclrtrsetts Cl.r-ricLrlLrm l-rl.rcrvorkr. firlfillr'q reclrrircrrcnrs of the
Educ.ttion Rclbrrl lrrv of lr)r)3" (MesslchLr\crr\ l)cpilnrrrcnt of hclucarion, 199g.
i. l).
Ihc dr ir,in.c firrce br..hincl nrilrrv srira(. reftrrr tffbrts rvorrld appeer to bc rhc colr
plinq of rct:rrds ot \iltctionr ro pcrldrntllcc. on rhe srltcn,rde rest. policvll.lkers ,trc
.lw.lr'( th:lr testitrq proqrxllls thet hlrve rltc grettcst ullp:lct on lhc curricLrluDr. rDr!rrrc
tion, end lelrning.Ic oncs thlt student\. !cJchet\, ldnri istrarors. p:lrellts, or thc
qencr:rl public pcrceive ;rs hevirq slncfi,ons or hiqh ltxkc,s il\sociatc(l wi!h tllenl
5 O,i..onr. Fvrl,)xti.f 101

a::i:::_ -:i!. ot (Madaus & I{ellaghan, 1992). In 1999. 33 of the Unitcd Stares had or shorrll' rvill
)ssessnrcnt :ls a
! :a: - ::.r:l the LtsL- of assess h:rve high sukes (c.g., high school graduarion. encling social prornotion) irltached to
:. -::t:: :o .enfalized high
0-
thcir tests. uhile 1.1 stlrcs Iirk rnoder':rtc'srakcs (e.g., a special diploma) to thcir
: l::r:,:i -\ R.iczek. 1996). assessnlerrt syslelns. Sanclions nl:ly iln'olr'e linancial considerations tbr districts.
i:: : .:'orgtniz;ttion of thc schools. or te:rchers. Sornctimes, howcver, the mere publicatioll of ourconrc inlor
: -,: : ::irii\-L.lv tltto orllous rutioli is considcred a sl)ction. There rvould appear lo be nvo principles uriclcrly-
:, ::i i::::a-nlJiItained schools ing the rrse ofsanctions- First. indivicluals and institlrtiorrs that are subject lo \]neliurr!
:::. : : ::ti aonsequcnt loss of \ril1 trke ilctiorr to obuirl rc$'irrcls:rnd:rvoicl purlishnlent. Sccond. lf informlrron on
r ::: ::-.-i .trosr' for netv con- or.rtcomes is trtought into the public clomain, principles of conpetitior. .vjll conre
::: .:--: r.::on .tnd contplilnce ]1rto oper:rtion. arrd. xs in lhe commercial rvor1c1. thosc that do rvell r,vill thrivc, thosc
: :::r , ::r.i .-r.rlurtion arranqe th:rt do poorly rvill rvith c-r' arval'.
::: . :::: :::sible. cost ellecrivc ORIGINS OF OUTCOME EVAIUATION

:::'::I: \er\'iccs itnd sociel The r.rtioralc firr, and pr-actice of. outcome evaluation owe x debt to el least sr{
: -::: : :roq_ing tlore rapiclly
of assessnlent in educrtior-t, school effec
soLlrccs: traditional cvalu;ltion. rratlitions
.: :.: Ljr gr!.Ltcr ctttaterLL\'. tivencss lnd cducarion productron function reselrch, dre perfornrance manageDlent
:r. i:r::: io bc continued
.tac'
nlovenlent, :rccounrrbiliry concerns. and technical devcloprrlcnts.
i:. -.,: l)urtn. Monnict., & Traditional Evaluation
A consider:rtion of the outconles of progrrnls is an intcglal fiertLre of many tradi
tioial approaches to evdullion, .rr1d, Lrp !o lhe 1970s. educational cvelurtiolls focused
: : :-i rre elaltution. C)nc is prim:rrilv on rssc'ssinq frogr:lnl outconles- The crtrphasis on outconles i5 1tro5t
: -. ::: rrociltecl u'ith org:r obliotlr in objectives-oricrltcd cvalu:rtion approaches. T,vlcr (1949). lor exarlple,
: ::- ::':-.:tous in lacking such focusecl or cducation:r1 objcctivcs ancl thcit rnclsurement ir the contcxr of cur
_---: t:.-:t,.ol. \\'L-rc
to sPeciii ricuiun evalu]tion. Orher approaches in thc !leriaD nrdition also accorded prorrri-
.- ' a,rrtltl. :lnL1 \\oul(1 ncocc' to rhe specjtcatlon of oblccti,.'cs and.Judgmeits of lhc c'xlcnl ro rvhich the,v
:-- :: - -.:t:L.ut (scc Schntidr. coulcl be saicl to ha\-e been achieved on the basis of progranl oLllcolne data (c.g.,
: --_ ::::r i,i orLlcorre cvaht, ProvLrs, 1971). Horvcvcr, these approaches diflerecl frorn rnarrv current outcome evrl-
-- , : :r:rrCr itrtpaCt tvhen uation ctlolts in linking prograln objectives lo lhe goals or objectivcs of individu:r1
: .. : ,::ithitrg otrtcomcs of sahooh or teachers rerhcr than to starcrvide curricululrr llrnrervorks, rvhile outconres
:: :-- -:: ::t:.\.trious \Lalcs hlve \\'erc not uscd for l-righ stakcs dccisions or t'br accountability purposes.
':. :: r: j:-_tt-_ -e.rrni g st:rnderds
Traditions of Assessment in Education
:- :...::: .,. hielcrnent related
:-: .:::i..lsileAssessrlent Feu' pcople would clisalirec u'irh thc vielr'' thrt lhc oLltconles of edtrc;ition ale
':: I i. ntc'iisures the pcr important. Hoq'cver, eqreelrtettt u'ould not bt' :rs rvidesprcacl on the rel-rlive irrrp,rr-
:-i: ::::,lltg srandlrds con taricc of outcorllcs, since inciividuals ciiffer irr therr Perceptiolls of the froniil(llar
:- _: ::: ::quiertrenh of the thar should bc giren to lhc varief\' of goals or oLrjeclives thal h:rve bccn posirecl for
::.:-.:: -_. ..: Eilucation. schoolnlg. Literacy anrl rturl)eracv skills lrc usuellv :lccorded pxrriculir lltti'orrancr.
1998.
artcl thc'use of irriornration on outconles to nrake decisions abour the eflictir'clless
of schools rnd teachcrs, besed on stuclents lcqr-lisilions of thcsc skills, rexches b:lck
:::- L. tu Lru fltc Lou-
.a-::: -: :iii. Policvnrakers arc: into the llsr centurl Pcrhaps thc best-kno$'n cxanrple\ of this lpproach ere
': - :: :'.r -_,,rrriculult, instruc pavnrent by resulls schernes t'hich were irttroclucecl irlto Br'ltish schools in 1862 to
l:a :::r::::!rIi. pilrcDts, or lhC help inrprove studcnts'llterxcy :rnd nlxthen tical skills ancl teachcr clltcierrcl', rvlile
:: -:::r.:: :.::oat.1!CCl rvirh thr:nt
:1t the sr rc tinre \aving nlouc\'. I1l thcsc schenres, thc allocillion of iirnds to schools
102 ll Qne\tjoir\./\,1ethor1s Ori.nr.d Errlurtior MoJelr

$'es linked to student\' achievements xs nrcasured bv written and oral e\:lnirrtruns


irr rc:dlng. *'ritins. and erithmerlc. Rcsponsibiliw fbr rire flilure of students rvas
pl,.rorr rl. lro rld r. of rrr.rrcr.
Clrorvrh in the use of stlncllrclized tesrins in this centur)', espcciallv in the Unitcd
Sutes, rcllc.ts continlrins iDtere\t il thc or.llconrcs ofeducation- Rice! (1897) rvork
on spelling is ln clrLv crarrplc of outcome ev:rlu:rtion. Lrfbrrratlor-i or outcornes,
of collrsc, has been used tbr e varicry of purposes, onl,v some of u,hich rclatcd to
thc evrluation of plogr'anrs or cvcrr of scLiools. Ti'sls rvere nlost fiequentlv used to
asscss rhe pedi)rmrlrice of rrrdiviclual stucients. Orr thc besis of their ralue in this
conrext, howr-ver, Colerun and Keru'eit (1972) proposcd that thcl coulcl also
be used to pro\,idc rncasules of school pcforlrltnce in e.,'aluating "eclucatiolt:rl
clvi onnrents."
Over thc pesr rhlee decades, st:rndrrdizcd tcsts hrr-e been used increesingl,v :Ls
instruments of national cclltcltion relolnr-Their use in diagnosirig rvhat rs rt rortq lr
educiltiolr, loslcther \'!irh the legislarivc :lftcntion $.hlch testins hns received, reflect
a funclamental sbift in thc olllcial educ:rtion rvorld, rot olly in thc purposc for
rvhich srandardizecl tesls art' uscd, but also in pemeptions of cluality rvhich have
rrovcd t-rorn a cr:rnsicleretion of school liciiitics. rcsorLr:es, end condirions to the ()ut
comes of schooling (Matlaus & R.rczek. 199(r). A rcccnt illustration of the extent
!o rvhich outcomes luvc bcconre i plonuncnt coDcern of polic,vmakcr's is to bc
founcl in Prcsidcrt Bushls Arneric:r 2000 proposel (US l)epartments of Educatiort
and Labor, 1993) th:t pavecl the rvay frrr the Eclucatc Arrrcrica Acr of 199-1-This
legislation proposed that nerv Anrcrlcan Achievemcnt Tests shoulcl fbrrl part of a
l5 point :rccourrrebiliw plckagc designccl to cllcoulxge parents, schools, end com
nruniries "to neasule rcsults. conpare rcsults. and insist on changc {'hcn the results
arcn't good enough" (Go:rls 20(X): Educatlon Arlerica Act. 1991). This lcgislacion
rva! never irriplcnrcrtctl lnd rhc idca oi l "voluntary" national te\t is still on hold.
Nonetheless, nlan,v strtes luve rdopted dre centrel icle:rs irr thc lcgislation in clcsign-
ing their or,r'r standir-cls brsL'd rcforrrr progranrs.

School Effectiveness and Education Production Function Research


A hrge rirLrrbcr ofsrudies ofschool cfli-ctlvcncss and ofeducarion prcdlrctioIr frlrc-
tion research has uscd nrclsLrrcs of eclucational oLltconles, usually standardizecl tests.
in rheir eflbrrs to cleterrlirte ch:r racter is ric s of eflective schools.An inpur output rep
resentation of schooling was the rrrodcl nlosl lrequenlly cmplo,vccl: studc'nt achieve
rrrcnt;lr a point in tilic u'as rclatecl to a scrics ofinpurs, usu:rl1,v iderrtiiied as lirrrily
and background influences, school resour-r:es, and school ch:11:lcteristics (..g.. currcnl
expenditure- tcacher qualificetiol]s and erperiencc, pupil te;rcher ralio) (see
Hanushek, 1997: Mrdns. Aiasian, & Kcllaghan, 1980).
In line rvith thi! traditlol), scveral approaches to oLltcome er''aluation collcct data
on irrput in an eilort to identi$' ficrors associated rvith studenr achievement. The
use of indic:rtols (rvhich nrighr bc dcscribccl as statistics u'ith evaluativc relevance)
in outconre evalu:rtion firs plrticularlv *'ell rvith thc input outPut conceptualiz:rtion
a (\rr.onic il!,trLurtion 103

::. -i ..rr:rl e\atliDations of schoolirrg. Rellecting lhe inpur-oulptlt rrotlcl, indiclrors ttscd b,v lhe Natiorial
:.: -::: iti students $'as Ccnter tor iclucation Sl:rtlsti:s of thc U.S. l)epartmellt of Education trorv iliclude
conrcxt and outcotlc cht:t (Steln, 1986) At rhe international levcl' OECI)
(1997)

in desclibirrg lhc ednciltiorr systetns of menlber colrntries hls' duling lhc 1990s'
.:-: .:::lill\ 1n the Unitcd alld econornic contcxt of cdu-
-,,:,- : il-Lies (1897) r'.'ork used inclicetJrs to clescrjbe tllc dernogt:rpbic. soci;rl.
in edtrcation, the learning enrirorl
: - :: :::::::.tlt on outcollcs, .ttior). dDancisl and huntln resources investcd
:, . :: : .-r: $hich related rcr Il1ent eild lhc organiz:ltion of schools, and stuclcnt :rchievcnlenl
::r: : : :: :icquerltly used to The Perforrnance Managernent Movement
.: a:.:. : :hcir- l'eluc in this ttlakers is rlol nc\\' in
r::-.rr:...ri thcv coulLl also Sensitivit,v to the nceds of plogr:rItr nlanagers and decision
cu"l,r,rti,rr,. Stulilebeam (1983), fol ex:rnrP1e. corlsidercd tliat the decision that h:rd
:: :: r'. :r'.rrtirlq'etlucatiotlel the key conccln ofthe cval
to be llrde. rathcr thlll progrlnl otrjectivcs, should be
uator. Clrrrent intcrest il thc use of cv:rluation frndirlgs for nl:rnagenient decisions
; : :_:-: -r\ad iltcreesinglY as rvhich hes its roots
a ::::.'ir:rg nh.rt is rvrorlg in has a rather difl:rent origitl. hou'evcl: perfornullce lnanagemetrl'
in *re 1930s but sre*' in populariq in thc late 198ils and in d.rc 1990s alongside
r . ::i.r:.: h.rr rccc'ivecl. rcflect rl:ul
--': : . irr rhe purPose for rnore cstablished evalr'latiorl :rppro:rches Whilc rhe gcner-al aims of performance
agcmeit "lo base;uclenrenrs of lhe eflectiveness of progrlttl eflblts on nlore appro-
::: ::. ,-: .1,.r:lin u'hich have
: :, :::-l ._!.n.liaiolts lo the out- p'ri"t. trust!!'orthY inlorllation, alld to jlllPrcve thcse etTorts" (Rlalock' 1999' p
",,d
:::: : : :ir:r.rooll of ihe e\tclll itS; d,, no, cliller fiom the ritrts of ntanv r-nole lraditional evaluation approachcs'
...u..prs,.,r,d"rlvi,te pcrfurnlance rlrenagclllen! difli'r frorrl such epprolchcs
in a
-.: :: :olrir tlr.ikers is to be
nulnber of lvlvs.
-: : i: r.:rl1fa!\ of Educetion Whi]c trlclitiollal evairrltion qle\\' out ofsocial sciencc resealch, adoPdng
ils L]asic
:: ::: :::..r \ct oi 199'1, Thrs
conccpts and techniques, perfornrancc Irlanagenlent has ils roots in :i-
bureaucrltic
..1 ;. 1, ,L.r111i ibrrn part of I ones relat-
cnvir;1n1enl. It is trased orl plalloirlg rnd rrrlln:lgernent ideas' particularly
_,:: : -i:_:i. iahools, and corn
,:-: ::: :l:.::rge lhen lhe rcsul$ ing to tlulliry lssLlL-Allce' cusl()lllcr satisf:ction. and conlinnous illrpr-ovcnlent ll
deter-
:-: ::: . r'.)+1. Ihls leeisllti(t) iniolves defirting perfotnlancc in ter s oftesults, setting pcrfornance tlrgets'
rnining thc.*t"rti to tnhi,,lt resr.ll$ lre acirievecl using perfotllrancc indic;rtors' and
_ :r:.:::: rrst is still orl hold
basir'rgl ..r.r,.,rc. alkrcalion decisions on perfbrmatlce inlorntation ltr 'Lrnl is to
::-: : :.: ..-siiiation in desigtt- limited nurnbcr of oulcorlle nlcasures
pror,ide r'.ipicl ancl continuous fccdback on a
th"t aa" p".a.iu"d to be of interest to policl'nrakcrs achninistretors stakcholdcrs'
polirici.rns, arld cuslolllcr\. arttl ro be ofvalue itr n king decisiolrs
::.::..: Research (Blalock' 19t)9;
I)avics, 1999). The tnertlger, not the "scierttlfic" polic-v analyst' is the charisrnatic:
: :: : -:- ::rrrn prorluctiorl func and thc achicr-enlcrlr of per-
figure; eiicienc,v lncl ccononrY are lhe ltrain conccrnsl
--:.. :.-:-:i i itrndardized lests, fuirur"rt." t..g.t, is thc sign of "rdminisuarive heafth' (l'}ollitt' 1993)'
: ,: i \n i.pul-output reP-
.---. :__i .i'. !'d: studenl acllievc lt rves ll rlis col)!e\l lhet rllaniger]lent illfor-lrration svsterrrs (M[S) grerv irr the
1980s, desigrrecl to spccifv llre slructtucs ancl procedutcs governing
the collection'
:::. :,-:r,-. rilcnti6cd ls fallilY The dcveloprrlert was'
analysis, pre-se ritarioll , errd use ofinft)rllrldon in organizatjons-
: :.r. -::::;a:L'ri\dcs (e.g.. cltrrent thc- growrh atrd irlcrcrsirlg tonr
:rr leasr in part. I responsc to lhc ,leed lo ilonilor
-. ::::: rr.raher mtlo) (sec decisions eborrt rcsoulce :rllocation ()utcome
arld to justi$
plexrr-r, of systenls
infbr-
r-:i : : r,. rlLr.lttotl collcct data
.""1,r.tio,t fits reldily into thls picture b,v providing relarivelv sirnplc statlstical
lration rLroul e or aclivir,v on :r tirllely basis' Whilc more or less
s,vslenl, proqrarll'
-::- ,:-:.i:trt Jahievencnt The it is llot the
ra:r: :::l l lluativc' relcvnrce) conrplc\ ellalYscs lll:ty lccoDlP:rnv this lllortnation in sorne cvaluadons'
prittrar-v purpose of oulconte evaluation to provide thelll'
: : :-:. r --:lLrt concePtualization
l0l JI ' 1'r ,,. \l 'l'. I I | . \,1 .

Acountability
In reccnr veers. ;rccotLntabilitv has acluevecl incr-eesing pronrinencc ir qovernment
icLnil)islratiols in nuny countrics. Melsures ro cortrol ho\r stikeholders djschargc
their obligations her-e bccn clcvised ls I rrrecLrnisrrr for cle:rling u,ith jssuc.s u,hich
rrisc lron I nrurbcr of phenorrcrr:t: incrcrsing dcnrand lor scr.vtccs couplecl rvirh
clinrinishing lesources; :r nruhiplic:rriol of rclbllr str:ltcgic\; \\'eak :ldtnlrtr,!r.ttl\c
inslrurroitsi lrrrl corrrpctinq velucs tncl clemands in plur-alist cukurcs. Thcsc rrrcl-
Jrrr-es, \hich hlve Lreen :rppliecl to n renqe ofpublic scrvices. nright sccltl r re:lson
:rblc \\'lt to bring order to corttplc\ irnd poorly ulclcrstoorl cnvironnlents. It i\
cnvislgctl rirrt intirrnr:rtion brsed on dre mc:rlures \vould leacl to rhe urc of eclnin-
istretive controls ovcr the Lrsc of inpurs to cr)slLrc lhat spcaificcl procedures :Lre conr
plicti rvith. it rright rlso sinrplv irtvoh.e the iclenrification ofproducrs rhlt rnect
13ut
.r speciliecl \trnderd rnLl prodrrcts thrr clo not. It ls rc-glrdc,:l:ls a relarivelv sinrplc
lrcl straightloru..trcl t.tsk to use detr fi-om en olltconle ev:rllr:ltior lo phac drc onus
for chrnqe rnd rdjustnrent or) thc person or jrstilrltiol) itlcltiiied as being lccourt-
rble, rnd to ph.L- orrL'ls rrll\t in the opcrition ot a competiti\.e nr:rrkct and the thre:rr
or pro]]risc ofsarrctions ro briDg abour rhe desired efli'cr. In thls siru:rtion, the otrus
is r)ot on:r nlrn:lqer to idertlfv riesir:rblc aspccts of irlrflcrr)entution ol conclltjolls
thet ncL'.1 to lrc chlrgcrl. He or shc doc! not have to trv ro undclsrirnd or t'xphrn
nh_v sourc inclividu:rls or instirrrtlons xle "cfli'crivc" lnd sonc irrc not. A1l that is
rleccs\i1rv is ro iclcntili thc cllcctivc ercl tlrc rrolcflectivt. arrcl to heve st:rrisrical dlte
to sllpport the judqnlent.
Despite problcnrs rssoci.rtccl \\'ith oLlt(:o1llc c\'.thriltjor considcrcd be1ou,', eccount
:lbilin i\srlcs loorn llrgc in considcrations ofschool refolm toda\i For L'xiljlplc. llrc
Erirrcltlorlrl Improvemcnt Acr edopted ill Tenncssce in 1991 ctcatcd the need to
specili rhe rncxns b,l' whiclr te:lchcrs, schools. arrcl scllool s,vstcms could bc hcld
,rcaourrt.rLrlc lbr rttccting oLrjcctilcs ser firr'leDtressee's educrtion st.stcrlts. Siticc tLrc
tircrrs r':rr on ploduct mthcr thlrr on proc_css. ill orltaonrcs-bascd ilssessrnent s,vstenl
r.,'ls cstabllshccl llld hls been enrbccltlecl in the Tennessee Velue Addcd Asscss lcrit
S,vstenr (lVAAS) rvhich lornrs :rn intesrrl p;lt of lcgisl;rtion (Santlers & Horn,
l9e1).

Technical Developments
Thc arrjl;rbllltr. of rchrirclv kxr, cost tccIroloqlcs uidr nrassivc col)puhrLli !.Lp.L
bilities Lus gleirtlv xirlcd the devcloprrrcnt. not onl) of lar'{lc sc:lle tcslinq progr:llls
lo obtijrr olrtaornc dilti, bLrt itlso of nr,1 rlcfrrft rll,)rnr.Lt]L'n \\'\tcfr\ rn gcncril end
logisticrl pl:rrrning. Outcorrrc evelurdor rs grcarlr'liciliratcd by thc- caprcrr t', rr,rrc
vust :uu()Llnts of d:Lta, () link clrre collccrccl :ll diflctcrt porrts ir) tiulc. .rrr,:1 to carrv
oLrt \{)phi\lic:1lcLl sr:ltisticrl :trl.rlvrc\.

THE USE OF OUTCOME EVALUATION


Thc tordr'rcv firr qor'crrrnrcnts to reke rcsporsibilin fbl qualiw by'sctting sttt,.lartls
rLnd morlitorirlq rchol:rstic :rchievcrnent. coupled with rn rllocxtion of responsibiliq'
i ()rnrf,. Fr,l,,:.i.n 105

for thc usc of lcsoulces/iJlputs ro proviclers, c:ln tre found in:r nide r:rnqe ofcoun
:: :: : : -_a lll qovernnleDt tries. This is a charrgc liorrr a situation rn which, Lrp to recenth', nroniloling arcl
.: :,::ir,rlJcrs discharge evarlu:lrion svstenls \l'erc nlore c()ncelned \\'ith re\oLrlces end inrplcncrit:ttiou tl-un
: ::-::: ...:rh issues rvhich $ith assc-ssir)SI rc'sults. Lr nrrn\' aountlies, aspacts of pcrforrrrarrce nersures rre nolv
: : : .::.ti:i corLplctl r,r'irh undeN'ritten b,r' leglshtior.
.:::r:,'atk .rdntinistrative Li lhc Unitccl Stetes. rhe (iovcrnurenr Pcrlirrrrarcc and l{csulrs Act (GI'RA) of
---.:.. -:::.irc'r. Thcsc tnca 1993 lrs implemcritcal ir) C)ctober 1t)97 ls a responsc to lcports oi *,astc end inef
:.-i:. :: :::tt scenl a teason frclcrcv in governrnent spenrline- Tb 1cslole public confidcnce in governnrerrr, all
r::.: . .:r.. irornlcllts. Il is fidelrl lgcrrcic-s rr'ould be heltl rccorlrrtilblc for rchit'ving progrlnr rcsults. scrvicc
:.::-r:.':lie Lrse oi adIrtill- queliry cus.onlel s:rdsfirctlorr, and ibr pr<)ricliDg Congress uith srrlllcient infbrrl:r
::--: r -- :rlracdules tre conl- llorr to inrprovc decision makinq. l)olbrnr.incc'rrc',rsulenrent rvorLlcl bc rctltLircd ard
,::: :: - -.rodLlcts rh:lt nreet the lesrrlting clltl rvorrlrl bc rladc'public.A renqc ,rf public:rtiorls proviclrng.r r:rr,,
::::: -i .r rcl.rtivclv siniplc nllc fbr. end descIiption ol pcrlirlnrrn.r ]lrcr\LncIrL-ft ( rrrf.urD:j l,.r r'csulrs"). as
:,:-::r :_. :o Pl:lce lhc onus rvell as expcricncc ir lts lnc- iras been preparecl b1' the U.S. (lenerel Accounring
:j:::.::..-i r< bcirrg ac,::orLnt ()ilice and orher:rqencies (http:/ /."vu ,,rr cc-r.rst]lr.gqytfatll!+talgprdi$.blut).
.:: . :: ::Let irnd the thrcitt Major ch;rrgr's heve occurred ir government rqer)cies fb1lo$'xrq the legislation.
.: , :::. .irulrion. the onus Fol eramplc. thc Urrircd Srrtcs Agencv firr lnternationel Developnrenr (USAII)) has
: : :::.::::.:tii{rt) ol co ditions developeci fil its F.rncieci plojects e 'rcsults fi.inrcrvort" rlhlch irrvolvcs spccihc:rtion
=. : -:rJ:rrrlnd or expiain
-i
of golls, objcctir'cs, irdicarors r'itlr periodic tarqet!, intelrrrediate results, :lnd long
, :: . :r.- r)or. AII that is ternl nct r-e,iu1$ (r'eprc'scrtting the effect of rhe interventioD) (Tbftb1oD Wciss,
: :::r:r-1\'c ir.ldstical data Bcrtrand. & Tc-rrcll. 199t)).
Ev:rluetion rcrivjtv outside tlie Unitcd Stltes is not well doclunented. However,
. : .::::r.1 L-elot. eccount- lL sccnrs relson:rble to sat' that the c\tL-nt or rengc of cv:rlr.rrtion rctivirics lbund in
: : ::: -:-:.. For elentplc', rhe thc Urrirctl Statcs is nor fbuntl clscrvherc, despitc r rccclt srugc oicvrhrrtron.rttiv
. -::.:rL.d rhc recd to it\", or nt lelst r recoqrrltiol) of its need. in menv countries. ln Sp:rin. lirr er:rnrple.
. ,::r:rs cou1c1 bc held govcrnnrenr hes rcsponcled to leeisliuiorr rccllliring r'vxlurdon lollo\\'in!a govcr rrl]rcnt
:: :. .. itl'nr\. Sincc the xction ir1 contraatirq sclvlccs. creatinq condition\ for conlpctltion, rnd rrisinlj the
_i :. :.:-:
.liic\srtreDl svste]]r issue of accountebilin- lhe responsc rcllccls l plefi'rcnce lirr eveluetion lpprolcht's
,,-: ::. l.lJcd Asscssnrent thrt lre cornpatiblc lith rhc production of nlan:lgenrelrt conlol indic:ltors and irre
:,. .: :: \.rtdels & I{oln, useflrl in inforninq decisrol nrking in rhc policv process. For e\rn1ple, the (l:lrihr
Hcalth Services Admini\trirtive ()t'Icc rrrorritors poprihdons scrvccl. c_ost, lnd oLrtputs
(e.g., nrrrrbcr of visirs pL'r' inhabitrnt per dr\'. lrLrrbcr' .lrt1 cost of prc'scriptions)
(J3all:rrt. 19t)8).
:a :: .. .:' aolllpuh[g Cl1pi1 LIsc of cvnlultiolr (throueh usnallv of .r rathcr olcl ftshionecl v:rrierv*) hxs rlso
=:-.-.-.-: tciting Plograllls
grorvn ir othcr courtrics clurinr: the 1t)8{)s lnd 1991)s. Lr I)cnnr;rrk, rredi-
r:rpi,:11,v

. ::i':t)\ irt {eticr]l and tionll cnrpiricrl nlerhodobgies (usuelll, survevs) to pr-ovide Lhta lbr politicll and
:r: i:rp:lciw o storc ori1anizilliorrl (lcvclofnrclt. .ontrol. lroritoring, rnd moLler niz:rti()n xrc lavored
: :irc'. .1nd lo clrrv (Hrnsson. lc)c)7). Lr Frlncc, -'rvitlespr-crd inirtLr:ltron with prrblic polic,v evrlu:rlio!r"
rs a nlc:lns of nrorielnizinq public service hrs bcen rcportcd (l)urarr. Morrrricr. &
Srrrith. 1995. p. .15). ln ltell: dem;urds to ploduce :rn evlrlulrtion firrnelork for- r'eceut
refi)rnrs iri lrerlth scrviccs (.d::iatLla li:ztt:iont' dtlla sa rd puhli,:a) havc resulted in ren
slons belrl'eell :rn rpprorch tocused orr nlilnaqL'rnerlt rDd one mor-e olienlecl to ctlcc-
ir :.'trirrq shnderds dveness ard quality rs\es\n1ent. Norrvll. also sccllrs to be showing signs of increlsine
:.::..:r oi responsibilitv enthusix\m ftrr evllultion. thouqh i\sues have nor vet cievelopecl with the shiirp ess
-
106 Il. Q!c5tioni/N1cth.d\ ().i.rrcd Frrlurnon \1o.1c1.

of fbcus observable in Anglo Saxor cor-rntries (see Nervs frorn the (lommuniry -Lrrrr/
rdliot, 19()8, 1,373 379).ln drc Russien Federation, thc requirement of e unilbrnr
curriculurn in schools is being rcplaccd by grcltcr ilutononrv fbr rcgional :ruthor'i-
tics and schools in conjunction rvlth ourcome based curriorh (Bakker, 1t)99).!(hile
the evaluation rmbitions of mrnv counblcs sL'crrr less rh:rn nlodest, rexlizetiolr is
being hanrpereci b_v lack oi data, expertise, instrurrrcnts. .iud drc iuG:lstrr.rclLuc
required f-or large-scalc data collection :rnd :rnelysis.This point has been nlrde regard
ine rhe developnrerlt of ev:rlu:rtion ir thc l)coplc'.s llepublic ofChina, where eval
urtion \\'as u kno\\1r Lrp to rht'carLv 1980s, but is ro\\'scc| to bc inrportant in the
conte\t of n:ltion:11 developrnelt and economic qro*'th. Mariy steps :rre beirrg Lrkerl
ro inrprovL'rhe colultrys evalu:rtion capacitv (Hong & Risr, 1997).
'We turn riorv to dcscr iprions of spccific olltconle evalu:ltion eflbrts in education
at srrte level (U.S.), n:rtional 1evel. ancl iutcrnltiolJ lcvcl.

Outcome Evaluation at State Level


In tht' United Stetes, stere ,:lcparnlents of education arc the nlrjoL- players in
ou!colrre evaluiitjorr, collcctirle dalr on studc' t Achievenlent, publishing rhe d:rt:r, and
allorving conrparisons to be nlrde bctuccr schools end school districts.
In Tcr.rs. for c)onlple. outcon)c clata ere providecl at all grlde levels for a r:rnge
of varilbles including lca,:lerttic rchievcnrt'nt, stLldcnt pKxrlotion rate, sturlent atren
d:rncc, dropour ratr'. pe!:cent:rge rek]ng the Scholastic Apritudc Ti:sts, and po!t-school
college enrolnent ri!c. Cash rc\\':rrds to schools rnd to indiviclurl professional staif
are gil-en rc schools that provicle test clata for 95 pcrccnr oi eligible studen$ rnd in
rvhich ar lcast half its cohorts pcrlorrn better than il llornl glor-Lp (Wcbsrcl, Mcndro,
& Alnaguer, 199.1).
The ltnnessee Value Addecl Asscsyrient Svsrcrn (TVAAS) is also ln oulconres
based svstern, irr rvhich thc locus of accountabiliry is ori rhe product ofthe educa-
tional experience, not the process. Thc TVAAS has been adoptetl end iegislated for
in statc' larv According to Jlr rllryr'r Pld, lir']-otn('!sct Stlnok 1993 oi the Stalc
Boarcl of Eclucatiol, "Stlte rnd local education policies rvil1 be focusecl on results:
Tennessee will have alsessrrent and nranagernen! infbrnution sy\tems that provide
ir]fbrrlrallon on studenrs, \chools, and school s)steiN to irnprove lc:uning encl essist
polic) rrlrkiri!5" (cited in Sanders & Horn, 199-1, p.301).Tcsting lirkcs placc at all
gradc levels in reacling. nathenetics, scicncc, llngu:lsc, encl social studies. -]ud!3ren!s
:rre llia(lc on thc basls of the dlta tlut are collected on the etTects of\chool s)sterrs.
inclivicluel schools. rnd iridir'ldu:r1 teachcrs. l)atr on the firsr !r'o rle released to dre
pr,rb1ic.

Outcome Evaluation at National Level


The nrost obvious eremphrs of outcome evaluetioD :u national level are "national
$'hich hxve opereted in thc United Kingcloru in o11c forlli or alrother
assessnrenls",
since 19.18, ili
rhc ljnitccl Slatcs sirce 19(r9,:rncl in France since l979.The United
Srates National Assessnent of Eclucational Progress (NAE[)) is rhc nrost rvidely
rcportcd asscsvncnt nroclcl in thc liter:rture. lt is arr ongoine surve]', mandated by
I

a Out.omc Evalurtion 107

ol district
:i-.-i - _:::1r: (-omnuniry, -tlal- thc U.S. Congress lnd irnplelnented by trained 6eld staff' usually 'school
educarional achlevelnents at
_j.-. ::.:-:::_:lcnr oi a urrifornt personnel. Thi survcy is designed to nrelsure students
g."d., and reports the percentage ofstudenls scorinll in the thrce
.i::: :: . :,\! regional authori_ lp.cifi.d
"ges "rr.l
a'or*ao""..i"t p.r{o_.rt ."tetoties: "bmic"' "prohcrcnr"' and "advanced" lt also
,--:--:,-: r Brkkcr. 1999).While "r'ta"
of subpopulations defned by demographic cb'rr't'leristics
--: :: ,-_. :tr.rJc.t. realiz:ltion is "r.rrriu.r ".hi""'.ntents C)\'er the Ye'ars' cletails of thc adrnrnistration
:::-:..::. :::i thc iDfrastructure ty ,p..tft. background
"rJNaEi' It".,c ch"rtgcd; for"*ptti"ttt
era''ple, in the tiequency of assessmenl :rnd itr t6c
l-:. : . ::_: :r:. l.een nrade regarcl- "f everv second yeat on
i-::-:-:-_ .r: Clhina, rvhcre eval grade lcvel targelerl. At present, assessnlents arc conducted
_.. :::: :!r 'oe rtrportant in the s'ar,rpl., of rt.til"rtt, irt glacles 4, 8' and 12 Eleven
instructional areas have been
and rvriting' math
..--- 1.1-:-. i.rpi:lle being tirken assessed periodically. Most recent repol$ have focused on rcaditig
have beetr teported by state'
,f -\ :- -r9:1. ernatics and science, histor.v, geography, and civics Data
':.!-::::.'r. edbrts in education gendcr, ethniciq'. type of comnrunitl" and region'
' Nr,iort"l are now fcature of other educaliolr systcms lhrough-
.... '
"rr"rrrtt"rt* in industridized "tln-v (e g Australia' Canada' Finland'
out lhe \&'orld, nor only countrics '
irr..", t."t"nA, thc Nerherlancls, Norn'a-v, Nerv Zelland' Unitcd Kingdom)
Sr'veclcn'

but also in developing countries (see Chinapah, 1997; Glcane,v &


Kellaghan' 1996)
,::: ::: rire nujor pl:rycrs in school
An assessment of suJents' first language :rncl nlathenalics at lhe
elelnentarY
-.::-.::.:. .ubliihing thc data, and
in sorne';rnti a second
::-: !:..r!ri Jii[]cts. lcvel is inclucled in all nationei assessmcnts Scieoce is included
nusic, and social studies in a srnall nurnber' h-t mosl countlies' drta
: :: ..,. :::Je levels for a range l"rgu"ga,
"rr, lor a sarnple of stucleDts at a particular age or grade lcvel' but itr sortre
: :: : ::!rn rrle. student atten- "r.'.o1i".,.,1
cor-rntries, a]l ,t''.l",]t, the relevant age or gradc ]evel :rre assessed (Kcllaglran &
. r-:::-:-r: Tesls. ancl posr school "r
: : ::::'.:lurl prof_essionll st:rff Grisav, 1995).
::.:: -: :ligil-.le str-ldents aDd in Outcorr-re Evaluation at International Level
:- : ::t. i:.rLro i\\rcbslef, Mendro. in lhat they involve mel_
Tnrernational :lssesslllenls ciillcr froru natioial assessrllents
severd countrics, usually sinul
1,. -> t, iti\o i11t outconles- sureutent of lhe outcontcs of cducatton s.vsterns in
j _ r rcn.rsly. Il.pr.renlativesfiom nnnY counlries (usua1ly ftorr resc:uch organizations)
.:r: .rodLlct ofthc educa lbc ilrstrurncnt
agree on an inslruilenl to asscss achievelnent ill
a curriculum arca'
:.. ::::i:d :rnd legislated for
of sNdents at a patticular age or grade
-.-..: ! :..,ii /991 oi the State is- adn-rinistered to a lePresenlalive sanrple
data are c;rrried out
level in each coultry'' and colnpar'live analyses of the
of interrrarional studies ower n;rtion:r1
:-t:::: ::-': :\'trerrrs that prcvide if<"tl"gtt,tn & Grisall 1995).Thc main advantage
is the comparatrve liamework they provicle in assessing
student achicve-
1r :: r::::!ir,i lcarning and msist ",r.rsit"nr. givc some indicarion of
: -_ T:.irng .irkes plrce at all ment and curricular provision lntcurational esscssmencs
where the students ln a colllllry sland relative to studcnts
in other countries They
co1rD]ron curriculunr arels difels
.
--:. -: -. --i.r1 str,rdies. Judgn-ients also shorv thc e{ten! to rvhich the trealmellt of
:: :-:: .::.ati ofschool systens, the approach in a given
.aror, ao,rutri"r, and' in particular, lhe extcnt to lvhich
-: :::: :,.. o .1te released to lhe idiosyncraric This ilrtbrrnation rnay lead a country
ro rel\se\s it\
.;t;;; "t.y
be
curriculum Poiicl'.
The International Association lor the Evaluation of Educational
Achievernent
stuclics alld has carried orrt a series of
:- :: ::r!':1.r1 level ar'c "national UEAI-ftr, p-"**a internatiollal assesslnellt ofco lrrrie5 sirlce
,::-:-:: : lir one forn or another .,.'ai", nr,.noo1 achicvenerrt, attitudes, and culricula in a variery
desigled lo
tO;1. attlough orte of IEA's primar-y fuirctions is to
conduct research
: i:=--.-. .:nie 1979. The United also.intended to have
.-' \ iIP i: the rnost u'idei-v i,.fro,t" una.-.tt".taing ofthc tducation^l p'oce's' studies were to policy-
]: -::::::L surver,, nuDdated bY i-." p.".ri."l and appliecl purpose: ro obtain information relevant
"
108 ll. Q!.ni(,irs'Xltrli.d\ ()ir.!ii.(l l1\.tlurrLo!r Mod.l'

tlt.rkioq rrrd edua.lrioDill fltrrltlnq iD th. ini(-rest oi iltlProvillg cLhlc,ltioIr svqrerlls


(HLrsin. 1967: Posrlcthrr ritc. 1()87)
Tir ri.rtc. rhc IEA lt.rs concluctcrl srudics oi lxthclDiltic\ rclliclerllr'rrt. s.itrr,.
eclrcveurcrrt, r'c.tdillg lJtcrecll \,riltelt cotllposiliorl' Erlglish rt I lbrcigll lilllglr:li-rc'
I-rclch rs r firreiqrr lerrgLrrqe- civic cdrtcrriorl. colllPr.rtcl\ in ctlucrtiorl rncl prcpri
mun childcrre. T-.'r'elr rrlcl pefterns o1- r' hic|r'rrrcnl h:l\'f bccn dt'scriberl end corll
p"r.l.1 ...lirt, coltnfilcs. So lrl;o h:rvc diflercnces in ilrtcrlcled 'lrrd ilrrflclnclllcd
currict,l,r .rDcl ln tbr. cour.se tnki e plttc-t.ns of.rrrrlcnts. A r',uietv of collcl,rtcs of
ruclrc'venrclt h.rs beclr idcnrilicd. incltlding \!LLLIcllls oppor tLlllitv <r ielrll' thc
i1nlorlr)t oi !illle :l \Lrirjcct r\ srLLdicd, the usc of c()ltlPr.rlcrs. ilIId lesotrr.L's il1 the
honrcs of strtclctrtr.

APPROACHES TN OU.I'CONIE EVALUATION

A vlrien of rpprorchcs. dcp.-ttdittg on lhc oLllLolllc to Lrc essc'ssccl hrr bccn rrscrl
it or-rtconrc e\lxllt:ttion. 1n erllulrions lD thc. ficlcl of cducillloll. .tssrr.trleut. ,rf
\tLlLlcnt ilchiL,\,c tent lls!rlll! iltvol\c rhe rd tiDish.rrioD oicesls,jl c:t:rnrirrrtlons. fhc
pcrlbrm:rrrccs of in.livrdLtll rrudcnll rllrv tllr'n bc' .rggregrt.'d to thc leu:l of thc
tcrclrcr. school. clist lct. strte, or c\'en r).itioll to ello\'-jud*nlcrlts to bc Itr:rrlc lltout
irahicrcrnclrt rt thc LlcsicLl lcvcl.
-hrclgnrertrs rtnv bc
ntrdr orl thc Lrrsrr oirr .rdjnsteLl r.'strlri. Irl flrltish lelque r:rblcs.
r1,.. perc"ur"gcs of strr,lcnrs iD schoolr l$,er.lcd \'ert tg gradcs on p11l)lr,
( \.rLllil1.L-

riors ("pcribrnr:urcc trblcs') hrve bccn publishe':l since 1992- l;r the Urrirccl Stltcs'
nlost stlle .lacottnlirbilin \\\lcrlls lll tlrc plrst corllparcti lcllools:rllrl:cbool tLstlicts
on the br\is oi unrrd-juste,-l oLrtconrc nrr.lsLrrcs ((iuskcv & Ki1'cr, I99ll) Sintillrl,r" irr
irrtcrnrtiorrei corrll.rl:ltl\'c strtclies. coutltlics ele rrllkccl oll lh' blsis of nnerljlLsted

This procedLrlc is pclhepr nor stuprisirlg if()rlt.olrlc clilluiltioll is concenlcd pri


lurril-\'$ith rlescription. rlot c\|hIt.ltrorl. $iih lha fr-()(lucl oitllc cduciltionrl c\lc
rlcncc. rtor thc |roccs\ 1\'\hiclt lr \\'.rs tchictcd._lhcrc is. lto\\,,:'r'cr'. concerrl rbottl
tllc c\teIt ro \hrch strch conrput.isorls:rrc f_rit. PlrtictLllrh if tvrlrrrrio'r resLrlts erc
Lr\eal ft)l ilcaoLrlltrbllit, pLrrposer'1lrr'issue lt strk.- is th:rt of rliltilgtlishirlg
betrvct'n
thc 'rrct' irrlprct oi I proulilllr \\'hich rePrcscrlts or.1!'olr's thrl 'r;c ':lilcctl-v :lttrib
urablc to the pr6grrrrl.lrld "glosi inrp:rcl tllich rcdccn rrl rtlditiorl ro ll(I irrlP'Lr
r'

InBucnccs other llt.lrl litc Progr,llll Ltcirlq rllonitoled llic distirlcrion is rcrclill illtrs
trltccl jrt rhe clse oi stuclcrrt tcllicveltlcr)r\. \\ltlch rre gencrrllr'rccogtlizcr:l rs l-cllect
rn!l .l vrlicn oirrtllucrlccs. irlclu.lillL gerlctic r'tldonrlrcnt. llchielcrllcnt ur rL iLrirrq
,ci,,r,,1.,t,,,1 thc rupfort rtrlal .lislst:lnac lhit sludcrrts rc'ci\'c:lt llolllc 'rrld in thc corrr
tunin. rll of $frrclt nt,tl bc i clcPerlrlcnt ofschool rnd tcrchcr inlltrcrlccs (Serldcrr
& Hor.D. | ()9,1I \Xlel.(cr ct rl.. L()tl-l). lfsruclerrts tiiicr 6orrr school to scl"rl irr rheir'
lerr,ls of :rchicvc|tctrt rtltcn entcr.i g .r st ltool. tceslrtcs ()1 :lb\olLrtc level'i of stlrdcnr
.rchieYcnrent .ll a littcr (l]ta lt)J-r rlot.ldcqtl.ltcl-v Ici'lect 't school\ suc'c\r lll
I r'r\illq
stuclenrs fiont thcil irlitr:rl cno.\ ler,cls. Hortcvcr. it seerrrs r.'rsorl.rble l() sr\' !h.ll
schools enrl irrsritrrrions tlrorrld be h!-l(l .rcroLlltt.tblc crtrl-r' ftrI tltirrgs thlt thct'cen bc
5. C)utcoDle Ev.rlurlon 109

:t :::-::-',t:tL aducatioll systenls expected to influcncc, not lbr dre characteristics srudents bring rvith thern rvhen
drey cornc to school (Wooclhousc & GoldsteiD, 1996).
j--:::-,:: -_: .ahie\-c1lIenl, science In line rvith this thinking, several attellpts have been lrladc to develoP slatistical
:-:r:,.: :: .t foreisn language. nlethodolo€;ies that \I'-ilJ pernrir an assessnleDt of lhe confibutions of schools lo
--:j:::i :: i'ju..1rion, f, d prepri- student developmerlt in situirlions in rvhich the noararldon assiqnnlent of students
.: :r]-.: r::'-- icscribed and conr- is assun-icd.These ncthodologies are bxsed on lwo concepts. OIlc rel:ltes lo "norllllrl"

=- :::::-::J Jnd inpienlented acadelnic proqlession, rvhlch is thc avcrage ProEircssion that students nlake fronl a
:::::. r, ', ::ien of correlates of given sterting poilt over a particular Pcriod in thc schooi systen (dcscribed as
i=::: _::,trmnin' to leirrll, tl1e' "expectcd" proeress).The orher is related to thc extent to s'hich individual students
-_._:::::::!. lnd resources in tl1e or groups of students (e.g., iD a cJass or school) cxceed or f:rll bclor"' lhat lver:lgc
progrcss in the spccified time period.The diflerence is rcllarded as representin€i the
value which a particr ar class ot scbool has "addcd" lo students'pro€lress.
Shtistical proccclures are usuxlly basecl ort nruftiple-reqrcssion analysis and involve
::r: :. :: li\c\Sed. haS been USed conlparinll actual studcnt outcorrcs widl expectations or Predictions determined
: : :: :iu;rrion. asscsgneDts of curpiricallv on the basis of relevant inputs (attenclutcc, gcnder, ethniciry earlicr
:.':r :: :;!t! or crltlinations. The aclievcnrenr). The rnosr sophisticated of these aPProaches use longitudind student
: .::r::riiJ to the level of the data. in r,vhich individual sltldctts'eailicr achievencnt scores xre In:ttched lvilh their
': .::=::i.ltti to bc trradc about latcr achieverncnr scores. ln the ltnDesscc Value Addcd Assessnrent Systeir, for
exairplc, estinuted student gain scores are aggrcgaled to the levels of leacher, school,
-::..- :' In Brirish lcague t:rbles, rnd system and are conlpaled rvith national Dornr laains, which each school is
:,--:--:
-:rla\ on public exanrina expectcd to achicvc. Schools u'ith scores icss than ttvo slandrrd deviations below
::,-. -'1. ln rhe Urrilcd States, the noml lllust show posirivc ptogrcss tll risk intcrvention by thc Srate (Sanders &
::-:: .-:coli and school disrricts Horn, 199'1).
-:r.. ,.. K:rcr. 1990). Sirnilarll'. in Problcms associatcd with the use of valuc rdded nteasulcs include inadequate
--i:r _ tnc basis of unad.iusrcd co\''eraqe of dre lchicvcmcnls of schools, which rna,v vlry by curliculuil area,
lladc leve1, rnd teacher; itrcomplete data lbr students arising 6onl absenrecisru L,r
:::: . . --::tion is concerned pri student turnover ratc; regrcssion to lhe nlean in st:rtistical analyses; problenN wilh
::-::-:-: ,)i the educational expe- rcliability of nreasures \-he1l lhe nunibcr of studeols in a school is sorall; and hor'v
::::: :). :ti\\e\_er. concern about to fictor in lhe contextuxl eI1;ct on lchievcmenl crealed b1' rhe abilit,r' level of
j-::_ .,r'. ri e\: urriorr rcsults arc stlrdentsin l school or class (Sandcrs & Horl, 199'1; School Curriculum atrd
,r :r-: :: .l jirirrsuishing betrveen Asscssnlent Authoritl 1994; Tymnx, 1995; Webster et al., 1994; Woodhouse &
-:-_ :--:. :lrrr :ue directly attrlb- Goldstcin, 1996).
:::-.: ::: .:.i.irion to nc! ilnpirct,
: :. --ir.:rrlion is reedily illus ISSUES IN OUTCOME EVALUATION

:: :::::.,. recognized as feflec!- Despite its popularin lhe use of outcolllc evaluatiotl gives rise to a seri-_s of issrlcs
1i_:-::.:. : _:tre\'erteJtl on enlerinq Frrst, since outcorrte evalLration rests prinarily on assl-llllpliolls related to planntng,
::-:: . : .!: :tlrnt!- and in the COln- incentives, accountability, ancl consutnerisrn, it is not likelv to lead to greltcr Lrnder
- -: :.:-_::r inlluences (Sanders standilg of rvhat goes on in progratls, or to an identilication of the fA.tors that
i=::::::: ,-_lt..ol to school in their allict outcomes (e.g., the relative contributions of tcachers, schools, and a v:rricry of
i *-:. :: :::rrlurc levels ofstudent other influences, rvithin a prograrn or oulslde it). However, many lvould regard
:-. - ''alrri iLrccels to lno\-firg progrcss in uridersrlnding "horv" rnd "wh)r" programs havc arl inlpact r\ ilnporllnl
:: ir:::ri :..1!on3ble to say that lor rell iorprovenen!. Second. and relatcd lo lhe first Pornt, is lhe issue of idcnri
: r:.. :r::ntngi thar thev can be flcation :utd specilicltiorr of thc responsibiliry of provitlcrs ind clietlts, particulariy
110 II Q!.rtions/Nl.rhods OricDrcd tlrlurtion Modch

in sltuations in rvhich rolcs rrtay be enrbiguous errr:1 rot clc-ar1v seprnted. Ho$: does
olre establish thal r plrticuhr ourconrc wAs- eve[ in p]rt, rl]leDable ft) rhe inllucncc
of :r persoD to t'horn rcsponsibilitv for it nr:r,v hrve been :rssigncd? For cratnple,
rvhilc' it ls rc'asonablc to ilsslLine thet a school end Lclchcrs bcar solre responsibiliry
for studerlt lchjeverJrerrt. do Do! stLldcrits :tnd p:rrerrts llso Lre:rr responsibiliry? If
thls js so, horv shoul,-1 responsibiliW benveen the perties bc apportloncd? Ald
should the alpportiollllrclrt be the sanrc lor all srudcnrs. in all circuntsLrrces, at l1l
:rge levels?
Thir-d. peribrnrance indicetor's ntay be used. recorcle,:l, lnd interpreted in vervinq
\vays, thus llivillg risc to prcblems of corlparabilitl. For exanrplc, a colc sct of lDcil-
sures clevelopecl tr1, e Fetler:rl ltllcr:tgerlcv'I:rsk Force lo Inonilor rnxrkel Progl:lnrs
in rhc Unitcd States rvas dcsigneci to lornr the basis ofst:rte levcl rn:rira!!c1lrc11t infor-
r tion lvstenr supporling perfor|ill1rce ll1onitolillq. Ilowever. since no stxre oper
ales a full_v irrrcgrated dita svslem serving nrultiple prograrrs, and slnce choice of
peforrDancc trrcasrrrcs djiler Aonr olc progrtnt to :lDother. clata rre nor directlv
comp:rr:rble (l3la1ock, 1999).
Folrrth, since nr ry outconrc evaluarions locus on a lintitc'd rirngc of L,utLuDlr\.
rhc daca thar art obtlincd llr:ty not acicquatclv rcflcct svslclr or progran goals and
objecrives.The lcnlpt:laion. of r:ourse. is to fucus on what is e.rsy to llrersurej but
lhls rlay be to the derrlnren! of imporunt obJecrives. Perrin (1c)9ll) rcrninds us rher
"rtnnv lctivities i| tLre public policy rcllur. by tbcir very llaturc, arc conplcx lDd
rnrangible rnd cannot be reducccl to :l nlrmericrl figure . . . What is nreasurecl, or
evc'n nreasueble. oftcn bears little resenrblance to lvhat i\ relevlrt" (pp. 373 373).
Houever, focusinla oll a linlitecl set of outconcs is likel-v ro rtrcal that othcr or-rt-
comes will be neglectccl in prograni illipler[elt:]tiol.
Fllrir, rvhen outconrc' c'vilLladon i\ :l\sociated with high srakes, nrc'c!ing thc
recluirerlrents of lrie:rsurinq ancl reporting tnay bccone rnorc ilnporirnt thin $hat
a program lr'?s desiqned to achicve. relulting in goal clisplacenent. ln educarion. lor
cvrnplc. rvhcrr .rsscssllrcl-it r'esults bcconrc drc goal ofinsrlncrion, thc tr'uc prrrpose
of the instructioul proces\ mxv bc subvcrtecl as goals arc lcoriented to neet or
erceed "stendards." Further, eflbrts to improve perforr[aice on the mealure clo not
ncccssirrll) rcsult in il]UrovcrleDt in the ilreas that proqrams \l'ere designed to
rclieve. When rlcctinlt slandarcls beconrcs thc basis fbr budgetary cleclsjols. thcrc
is fhc furrher coDsccllrence rhal ploqrarns thilt ilec! starrclerds, ratlicr th:rn progralD
goels. ma,v bc co|tintLc'd. rvhiLc pKrqranrs that nreet goals, but Dot standards, rnay be
discontinued.
Slxdr, rvhc'n ev:lluarions are bascd on plecletcrllilcLl objecrivcs or srantlards, it is
unlikcl,v that unirrendell or uranricip:rted consequcr.cs rvlll bc clcrcctcd. Scvenrh,
the interpretatjon of drta itr outcome evalultions nliry not adequately acknowledge
diversities in the environrlent in rvhich progranrs u,'ere implenrerlred. lt nre-v rvell
be that:r proeram is "successftrl" in one coDtext, bu! no! irr anothcr. FinaiJy, drc cost
ofoutcol[e eva]uadorl rnav clivert funds fiom other tleeds, a not uninportant con
sideration at a time of r-esource corstrai)ts (Baftisdch er al-, 1999: Ulalock. 11)99:
I)avics. 1999; N.rrricllo, 199t1; Perrin, 1t)98).
5 ()uti!ne Evrlu?tio. 111

OF EVALUATION
OUTCOME EVALUATION AND OTHER FORMS
:: : ::: i::lr.ltell. HOrv dOes ht amotrg traditional
::-- ::--::: :al. ro thc influence ln conclusion, rve tray ask:'Wherc does oulcome er'aluation
tiom three not
: ::: :::r--Tred? For exenple, ;;o;;-*t to'proslr:ulr evaluarion? The question nray be addresed
-exci.rsive
:::.::- ..:: rone lcsponsibility nt,.,**it,u points of vierv: the context in *'hich an evrluation is
"irli."iy process rnd decision
:= -. ::-,.r rcsponsibility? If ;;;;i..i .tr, its ,ticti,odologl; ancl its relarionship ro the poiicy
:r---:, ::' lFPorliolled? And nakirg.
:::: : .- :trclrntstanccs, lt all Context
as it hes recelltly developed'
:::. :::-i :::cerpreted in varYing As far as context is concerned, oLrtcone ev'lultion'
Iays' frtting rnolc conlfortablv
t:: :::,::-:i:- r core set ofntea- ,liftt., frott, traditionai approlches in a nunrber of
eval'atior approrch. First it
: :: : i:_r:or rll.rrket progranrs ."iri ir, u*""g.r'"1 rnre;enrs thar rvith any program in theorY f,t
knorviedge that
:::-:= -.. c] nlanlgenlent infor- ,.na, a t parr of :, b.trcattcratic routine' providing
"
**, i, ."l.u,trt, ,,t pohc,v Second, it frecluently involves accounlabiliry consid
i: ._. ::- sincc no slale oper
.-.
^"y rcporting of per{ormance
::::::-:_...rnrl since choice of .'*tinn,. ."l"ti,tg lo tha scrulinizetion of programs and
ev:rlr-retion is in tbe contcrt ofvery
indicators. Thircl, the nost conlnlon use ofsuch
=-:::::. Jat.r are not directllt broacl and compiex Prosrans (represeoted in'
for example' all rhc etTorts nrade by
yeers) r:rther lhitn nlore discrete and
: r :::_:::i r;1nqc of oLrtcomes, a school or school svstelll ovet e number of
as ntost conlnoniy
::,:-:::: trr Prograrn goals and ,,,l,r." al.".ly speciftecl prograns Fourth, outcolue eval[ation'
or expcrimental pro
:- ,:.a:: -i aui\ !o nleasure, but or"a,ir.,l, r.i",., ,,, ,.rtt jorrtg p,".,tae ralher lilan !o illnovativc
,o -,aa"..*, i"t'"r or econonric problcms' Thus' it is
.'ot 'ormally
: --i:rrt:: 1r)98) renrinds us that ;;"r*';:";; progr'ln evaltr'LtiL'n i\' nol
isso.iat.d..,itlt triai runs ofnerv progrrnrs, as rraditional
-- i::. : :i-rrc. ,lrc corllPiex and inplelrlenta
:,--::. \\_hrt is nreasured. or ,."u cor.bi.ecl *,ith qu,,litaiive approaches to itssess progfarr
".r"*1,
tion and irnPact
:
-: :, ::l;'r ani' (pp. 373 373).
:i: .-_ neln th:tt orher out- MethodologY
a
The tnethodologies of outconle cvaluatiorl bavc some
afliliry rvrth (lrl\ (Lqbfl\)
:::-. -:::-, lr:ekes. rlreeting the on Popperian logical positivism'
::r: :: ::: irllportant than what npp.o".lt",, r'hich rvere largely based
".r"i.r",i,:,,r-t
ernploying qutntitallve ,,,"",ot"'' chains' ancl xsPlralions tolvards general-
d"tloJti*
. ::.: .::::::nt. ln educalton, for
izlrion.while outconre inciicators in themselves rvill ot
provide valid causal knor'vl-
:: :,-.:-_::r.n, rhe true purposc evidelced in cfforts to identi$'
.'aga inr.r"* in causaliry associatecl with thcil use is
::=0. - : ::arriil)led to nleet ()r utrderlying thc usc of addcd value
:--::.. : r:r rhe ncasure do not coirelares of;rchievc.rc.r and in the assrtt.ptio.s
'-: :.:::i;rr \\'ere designcd to techniques.
point ro an afnniry with tradi
. ::: :-::iien decisiorrs, there Whjie thesc aspecls of oulconle evalua[on ll1ay
there are also indicatrotts that
: ._,.:-il::r. nther thrrr Prograln tional viervs of e.raluat""i end indced of research'
is perceived as :r genre that is distincl trolll tradidonal evalua
i: -i -: r10r standards, mly be .r.rr.ont"
-- "u,,luatiort conclusion seelrs warranted t'hen onc
tion (see Blalock, 1999; Pollitt, 1993)'This and ioter-
consiiers that outcolne lllorlitoring in n:rtional
i'"ptt'""t"d aisessncnts
:: -,.',_:]\_ei or standards, il is x bti'lg ittlrttot"tl by gover nmenr' :rncl international
:r-:, .r, aa detected. Sevenlh, ."",p-r*e studies)
"",r"."i rf,. ,",,t" titt.tt *rote trrdrrronrl apprcaches lo
::. _ . :;.'qu;lelv acknorvledge ";;;;t;, "', ""d "ti.-ptnd'tttll ''f'
cialuation (sec, e-g, European Ciornmission' 1997)
ii-:: r::r:-i'ntenred, It Dray rvcll
: ::: :: ::r!rr:hcr. Finally, the cost Policy and Decision Making
::::::. : :ror Lrrlinrportant con rhe usc of orttcone evalua
'::i- :: :.. . 1999: Blalock, 1999; At this stage, tl-rcre is iittle docurlentlrtion arailable on
tiotrs in a poiic-v coI1!c\t The extellt to which
inforttlation dcrivcd liom such evei-
112 Il O,,.rfor\/N1.rh.J\ ().i.nri i,rtrlrrin(,, M.dcl\

u:rtions enters thc policl lren:r rvill no doubt cliller iionr countrv ao countr\',
depending on a coLrntrv's traditiors oi llovcrrnncrlL end of poJicr' antl decision
i killlj, JS wcll as on rhe relationships t'hich luvc .lrc:rdy bccn cslrblishcd Lrctw!-cn
polic\uukers. decisiorr rnrkers, and ev:llLr:rtors. lnsot:rr as dre methociologr' of
outcorne evxlLrrtions seenrs r:losc to rhat illyolved in errrpiricll qLr llrrl-rhrc
epproaches. rvith thclr rrlion:rl view of the policl proccss. onc might cxpcct
outcorne infbrmarion to bc consiclcrccl !-xoqenoLrs lo lhe proces\, pror-iciing "oblec:
tive", "ncutral".:rnd epolitical l:rlorrrrltror to bL- use.l instr unrenrell,v in policl'anrl
decision making. Irr thls vicu: as in earlv evaluiltiolr cflbrts, thc c'v.rluitor hrs r role
ro plu rrr resoh'rng polic-v issues. but not ;rs a plaver in the actu:rl policv ptocess
(Rrdre1li & I)crtc, 1996). This conclusion is reinforcctl \vhcn \ic consiclcr thL'
nurnber of outcor[e evarlurtion pro]L'cls iD lvhich rhere often is no iclentif_rab1e "evrl
u:rtor." Indccd, thc lcrm e\-aluillion often does not haYc a proDrincnt placc in dis
courses on lhe activides of rvhet rvc' alc c.r11ing outconle ernlualioir.
This shorrid not sLrlprise rLs. grven the lirrrited rrr-rnbcr of goals of rrrlbr r,r.rrror
r

productiorr thlt arc consldclcd relevent to oLrtc(xlle evahr:ition. Of the sir go:rls iclen
tified b,v Bl.r1ock (1999) that nrorc convcntion:rl nlethods of evxjlretiorl \tlrt to
nrccr. outcollrc evalu:lti()n is ljkcl\.to addr-ess orrlv onc: ,:lctcrrninlng iia progrrnr\
outcornes for clicnts (and pcrhaps its ner inlprct) rle codsi\rent rvith dr-sircd out-
conres :lncl lo improve there outconrcs. ()ulcome evaluatioD is not likel,v to provi(1e
infornratiol on Blalockls t'ivc other gorls: $hethcr or not a prognlr).s irrtcncntrons
are as intended; r'hc'thcr r progr:rn is beiug delivered ro the iitended trrllct pop-
ulation; \'uhcther r prollr:rm is bcine inrplcnrcnred as intcndecl; identificrtion of the
rDljor influc|ccs shepine e progrrllis oulco res: or rhc ipproprielcncss, ulilit\', and
\ocietal vrlue ofpolicics o \rhich x progrxn is brsecl.
Thc u':Lf in which outcome cvaluirlior irlbrnration is predicteci to \\''()l k in sonle
svstems suqllesls lhat thc cilbrt to eccdrlrnoclitte dre inlor nrlliorr in policv r,r.ill bc
slight. lf, iirr example. the prinrc plrrlosc of provicling outcclrrle infolrrr.rtiun L,1r
school pcforrnancc ls ro ltrach to it re\\'alds or pllnishnrc'nls lbr school districrs.
sch()ols. or terc|els, lhcrr tber-e rvould seem ro be little nc.e,:l to rellec! on, ol lry
to understand. horv schools t'ilnctiorl, or $'h:rt it i\ etrout proqrrns that ficilit:rtcs
studclrt !jro\\th. Perheps, the LlLlcstiors r:riscd bv these issues rre too demrnding and
ch:rllenging for a brrsv adnrluistrator. lhe e:rsier corusc is ro irnport lnarket modeh
and lcavc it ro conlpetition and consurner choice to bririg lboLrt dcsitctl tcfbrnr.
Horvevcr. ls long as this:lpproach is fblloucrl, rnlnv questions thit li:ive tr:]didon-
allv occupiecl evitlultols \\ill rern:lin Lrnanswclccl: doL's r ploqranr contlibute to
inprovcnrcnr. is it ecluitable. rvhet are the unintendccl colscqrierraes. and et t'h:rt
cost i! chlllgc achle\,-ed?

Anda mungkin juga menyukai