Int. J.

Social Research Methodology 11 Routledge

Vol. 8, No. 1, February 2005, pp. 19-32 ^^ T , . o, . . . n c , , o
Scoping Studies: Towards a
Methodological Framework
Hilary Arksey & Lisa O'Malley
Received 10 September 2002; accepted 11 March 2003
This paper focuses on scoping studies, an approach to reviewing the literature which to date
has received little attention in the research methods literature. We distinguish between
different types of scoping studies and indicate where these stand in relation to full system-
atic reviews. We outline a framework for conducting a scoping study based on our recent
experiences of reviewing the literature on services for carers for people with mental health
problems. Where appropriate, our approach to scoping the field is contrasted with the
procedures followed in systematic reviews. We emphasize how including a consultation
exercise in this sort of study may enhance the results, making them more useful to policy
makers, practitioners and service users. Finally, we consider the advantages and limitations
of the approach and suggest that a wider debate is called for about the role of the scoping
study in relation to other types of literature reviews.
Introduction
As t he drive t owards eviden c e-based prac tic e has gathered pac e, in c reasin g n u mbe rs of
systematic reviews report in g on t he effec tiven ess of t re at me n t s an d proc e dure s have
been publ ished by, for example, t he Coc hran e Col l aborat ion , an in t e rn at ion al body
support e d in t he UK by t he UK Coc hran e Cen t re based in Oxford, an d t he NHS Cen t re
for Reviews an d Dissemin at ion (CRD) at t he Un iversity of York. T he met hodol ogy for
c on duc t in g full systematic reviews in t he area of health c are, educ at ion an d c rimin al
Hilary Arksey is Researc h Fellow in the Soc ial Polic y Researc h Un it at the Un iversity of York, Heslin gton , York
YOlO 5DD, UK. T el: +44 (0)1904 321950; Fax: +44 (0)1904 321953; Email: ha4@york. ac . uk. Her researc h in ter-
ests in c lude in formal c are, employmen t an d disability an d researc h methods. Her most rec en t book (c o-authored
with Peter Kn ight) is Interviewing for social scientists: an introductory resource with examples (Sage, 1999).
Lisa O'Malley is a Researc h Fellow in the Cen tre for Housin g Polic y at the Un iversity of York, Heslin gton , York
YOlO 5DD; UK. T el: +44 (0)1904 321480; Fax: +44 (0)1904 321481; Email: Ijgl04@york. ac . uk. Her researc h
in terests in c lude researc h methods; the In tern et, health an d soc ial polic y; an d the volun tary sec tor.
ISSN 1364-5579 (prin t)/ISSN 1464-5300 (on lin e) 2005 T aylor & Fran c is Group Ltd
DOI: 10. 1080/1364557032000119616
20 H. Arksey & L O'Malley
justic e has progressed c on siderably, an d guidelin es for t hose c on duc t in g reviews are
n ow available (CCEPP, 1996; CRD, 2001). Curren t l y, t ec hn iques are bein g developed
within the soc ial polic y an d soc ial c are field by organ izat ion s suc h as t he Campbell
Col l aborat ion , t he Eviden c e for Polic y an d Prac tic e In format ion an d Co -o rd in at in g
Cen t re (EPPI Ce n t re ), an d t he ESRC UK Cen t re for Eviden c e Based Polic y an d Prac tic e
(EBPP).
T his rapid groMi:h in un de rt akin g reviews of t he l it erat ure has resulted in a pl et hora
of t ermin ol ogy t o desc ribe approac hes t hat , despite t heir differen t n ames, share c ertain
essen tial c harac teristic s, n amely, c ollec tin g, evaluatin g an d presen t in g the available
researc h eviden c e. T he followin g lists some of t he labels in c urre n t usage: (full) system-
atic review; meta-an alysis; rapid review; (t radit ion al ) l it erat ure review; n arrat ive
review; researc h syn thesis; an d st ruc t ured review. T here do n o t appear t o be an y
c on sisten t defin ition s of these differen t review 'an imal s', with the result t hat researc h-
ers may use labels loosely. For in st an c e, t here is a risk t hat reviews defin ed by t heir
aut hors as 'syst emat ic ' may n o t all adopt the same high st an dards in t erms of prot ec t ion
again st bias an d t he quality assessmen t for the selec tion of primary researc h. On this
basis the c orrec t label would be 'l it erat ure review' an d n ot 'systematic review'.
T he 'sc opin g' st udy c omprises a further type of l it erat ure review, yet un til rec en tly
muc h less emphasis has been plac ed on t he sc opin g st udy as a t e c hn ique t o ' map' rele-
van t l it erat ure in t he field of in terest. So what might we c on sider t o be t he main differ-
en c es between a systematic review an d a sc opin g study? First, a systematic review might
typic ally foc us on a well-defin ed quest ion where appropriat e st udy design s c an be iden -
tified in advan c e, whilst a sc opin g st udy t en ds t o address broader topic s where man y
differen t st udy design s might be applic able. Sec on d, t he systematic review aims t o
provide an swers t o quest ion s from a relatively n arrow ran ge of quality assessed studies,
whilst a sc opin g st udy is less likely t o seek t o address very spec ific researc h quest ion s
n or, c on sequen tly, t o assess the quality of in c luded st udies.
It is o u r c on t e n t ion t hat greater c larity regardin g the t ermin ol ogy an d me t hods t hat
su rro u n d l it erat ure reviews will assist researc hers in iden tifyin g when an d how suc h
reviews might be u n d e rt ake n . Whilst c ritic isms have been levied at bot h 't rad it io n al '
an d 'syst emat ic ' review me t hods we c on t e n d t hat t here is n o sin gle 'ideal t ype' of liter-
at ure review, bu t rat her t hat all l it erat ure review me t ho d s offer a set of tools t hat
researc hers n eed t o use appropriat el y. T o t hat en d t he sc opin g st udy is on e me t hod
amon gst man y t hat might be used t o review l it erat ure. Ou r framework aims t o c larify
when a sc opin g st udy might be an appropriat e me t ho d t o ad o pt an d how we might go
about un de rt akin g this kin d of l it erat ure review.
T o dat e, little in format ion is available about how t o un dert ake a sc opin g study, as
t hose sc opin g reviews t hat have been c on duc t ed t en d n o t t o provide detailed in forma-
tion (Hagell & Bourke Dowlin g, 1999; Jepson , Blasi, Wright , & Riet, 2001). T his paper
at t e mpt s t o address the c urre n t gap in kn owledge about sc opin g st udies. Where appro -
priat e, c omparison s are made with systematic review me t hods. We provide a model for
those wishin g t o sc ope t he field of in t erest based on o u r experien c es of sc opin g
published an d un publ ished l it erat ure for a st udy of servic es t o support c arers for people
with men tal health probl ems.
International Journal of Social Research Methodology 21
T he paper is organ ized as follows. First, we presen t an overview of sc opin g st udies,
c on t rast in g this approac h to reviewin g t he l it erat ure with t hat of systematic reviews.
We go on to out l in e t he differen t stages of a framework for a sc opin g study, in c l udin g
disc ussion of t he advan tages of in c l udin g a c on sul t at ion exerc ise. We c on c l ude by
explorin g some of the advan tages an d l imit at ion s of t he sc opin g st udy approac h t o
reviewin g the l it erat ure.
What is a Scoping Study?
Defin ition s of sc opin g studies are few an d far between . At a gen eral level, sc opin g st ud-
ies might 'aim to map rapidly t he key c on c epts u n d e rpin n in g a researc h area an d the
main sourc es an d types of eviden c e available, an d c an be un dert aken as st an d-al on e
projec ts in their own right, espec ially where an area is c ompl ex or has n ot been
reviewed c omprehen sively before' (Mays, Robert s, & Popay, 2001, p. 194; emphasis in
origin al ).
Whilst this defin ition draws at t e n t ion t o t he n eed for c omprehen sive c overage
(breadt h) of the available l it erat ure, t here may be quit e differen t degrees of dept h
(amo u n t of in format ion extrac ted from studies an d subsequen t l y report ed) c overed in
differen t kin ds of sc opin g study. T he exten t t o whic h a sc opin g st udy seeks t o provide
in -dept h c overage of available l it erat ure depen ds on t he pu rpo se of t he review itself. It
is possible t o iden tify at least four c o mmo n reason s why a sc opin g st udy might be
un dert aken :
1. T o examin e t he exten t, ran ge an d n at ure of researc h ac tivity: t his type of rapid
review might n ot desc ribe researc h fin din gs in an y detail bu t is a useful way of
mappin g fields of st udy where it is diffic ult t o visualize the ran ge of material t hat
might be available.
2. T o det ermin e the value of un de rt akin g a full systematic review: in these c ases a
prel imin ary mappin g of the l it erat ure might be un de rt ake n t o iden tify whet her or
n ot a full systematic review is feasible (does an y l it erat ure exist?) or relevan t (have
systematic reviews already been c on duc t ed?) an d t he pot en t ial c osts of c on duc t in g
a full systematic review.
3. T o summarize an d dissemin at e researc h fin din gs: t his kin d of sc opin g st udy might
desc ribe in more detail t he fin din gs an d ran ge of researc h in part ic ul ar areas of
study, thereby providin g a mec han ism for summarizin g an d dissemin at in g researc h
fin din gs t o polic y makers, prac t it ion ers an d c on sumers who might otherwise lac k
t ime or resourc es t o un dert ake suc h work themselves (An t man , Lau, Kupein ic k,
Mosteller, & Chal mers, 1992).
4. T o iden tify researc h gaps in the existin g l it erat ure: this type of sc opin g st udy takes
the proc ess of dissemin at ion on e step further by drawin g c on c l usion s from existin g
l it erat ure regardin g t he overall state of researc h ac tivity. Spec ific ally design ed t o
iden tify gaps in t he eviden c e base where n o researc h has been c on duc t e d, the st udy
may also summarize an d dissemin at e researc h fin din gs as well as iden tify t he rele-
van c e of full systematic review in spec ific areas of in quiry. However, it is impo rt an t
22 H. Arksey & L O'Malley
t o n ot e t hat iden tifyin g gaps in t he l it erat ure t hro u g h a sc opin g st udy will n ot n ec es-
sarily iden tify researc h gaps where t he researc h itself is of poor quality sin c e quality
assessmen t does n ot form part of t he sc opin g st udy remit .
Gen erally speakin g, these four types suggest two differen t ways of t hin kin g about t he
role or purpose of a sc opin g study: t he first two suggest t hat t he sc opin g st udy might be
perc eived as on e part of an on goin g proc ess of reviewin g, t he ul t imat e aim of whic h is
t o pro d u c e a full systematic review. T he sec on d two types suggest t hat t he sc opin g st udy
might be c on c eived as a me t ho d in its own right l eadin g t o the publ ic at ion an d
dissemin at ion of researc h fin din gs in a part ic ul ar field of en quiry. T he aim of iden tify-
in g gaps in t he existin g eviden c e base is c learly impo rt an t , an d may or may n ot lead ul t i-
mately t o a full systematic review.
T he remain der of t his paper is c on c ern ed with t he fourth type of sc opin g study,
aimed at iden tifyin g gaps in t he existin g researc h l it erat ure. We presen t on e me t ho d o -
logic al framework based on o u r own experien c es of c on duc t in g this sort of sc opin g
study; where appropriat e , we iden tify how t he proc esses we adopt ed might differ from
proc edures followed for a systematic review.
Methodological Framework
Ou r framework for c on duc t in g a sc opin g st udy is u n d e rpin n e d by t he view uphel d by
pro po n e n t s of systematic reviews t hat t he me t ho d s used t hro u g ho u t t he differen t stages
are c on duc t ed in a rigorous an d t ran spare n t way (CRD, 2001; Mays et al. , 2001). T he
proc ess shoul d be d o c u me n t e d in suffic ien t detail t o en able t he st udy t o be replic ated
by ot hers. T his explic it approac h in c reases the reliability of t he fin din gs, an d respon ds
t o an y suggestion t hat t he st udy lac ks met hodol ogic al rigour (Mays et al. , 2001).
T he me t ho d adopt ed for iden tifyin g l it erat ure in a sc opin g st udy n eeds t o ac hieve
in -de pt h an d broad results. Rather t han bein g guided by a highly foc ussed researc h
quest ion t hat len ds itself t o searc hin g for part ic ul ar st udy design s (as might be the c ase
in a systematic review), t he sc opin g st udy me t ho d is guided by a re qu ire me n t t o iden -
tify all relevan t l it erat ure regardless of st udy design . It is likely t hat as familiarity with
t he l it erat ure is in c reased, researc hers will wan t t o redefin e searc h t erms an d un de rt ake
mo re sen sitive searc hes of the l it erat ure. T o t his en d, t he researc her may n ot wish t o
plac e stric t l imit at ion s on searc h t e rms, iden tific ation of relevan t st udies, or st udy
selec tion at t he out set . T he proc ess is n o t lin ear bu t iterative, requirin g researc hers to
en gage with eac h stage in a reflexive way an d, where n ec essary, repeat steps t o en sure
t hat t he l it erat ure is c overed in a c omprehen sive way.
Wit h these differen c es in min d , we n ow go on t o desc ribe t he stages of the frame-
work we adopt ed for c on duc t in g a sc opin g study:
Stage 1: iden tifyin g t he researc h quest ion
Stage 2: iden tifyin g relevan t studies
Stage 3: st udy selec tion
Stage 4: c hart in g t he data
Stage 5: c ollatin g, summarizin g an d report in g t he results
International Journal of Social Research Methodology 23
An addit ion al , parallel elemen t is also desc ribed regardin g t he use of a 'c on sul t at ion
exerc ise' t o in form an d validate fin din gs from t he main sc opin g review. Whil st
c on sul t at ion might be viewed as an opt ion al c o mpo n e n t of t he sc opin g st udy frame-
work, it greatly en han c ed o u r work, a view c on firmed by ot her researc hers (Oliver,
2001).
Framework Stage 1: Identifying the Research Question
As with systematic reviews, t he st art in g poin t is t o iden tify t he researc h quest ion t o be
addressed as this guides t he way t hat searc h strategies are buil t . T hus it is impo rt an t t o
c on sider whic h aspec ts or 'fac ets' (CRD, 2001) of t he researc h quest ion are partic ularly
impo rt an t , for example, the st udy po pu l at io n , in t erven t ion s or out c ome s.
Ou r researc h quest ion was: What is known from the existing literature about the effec-
tiveness and cost-effectiveness of services to support carers of people with mental health
problems? We were aware t hat 'servic es t o su ppo rt c arers' was an ambiguous t e rm t hat
c ould in c l ude possible ben efits derivin g from servic es direc t ed t oward c are rec ipien t s,
suc h as day c are, for example. We also had t o de t e rmin e what illn esses were t o be
in c l uded in the t erm 'me n t al health probl e ms'.
Defin in g these kin ds of paramet ers, an d c on siderin g t he impl ic at ion s of adopt in g
part ic ul ar posit ion s, is impo rt an t at t he out set of a sc opin g study. Very wide defin ition s
of what might c on st it ut e servic es for c arers, for example, might reduc e t he likelihood
of missin g relevan t artic les, bu t c ould also gen erate an un man ageabl y large n u mbe r of
referen c es. Ou r re c o mme n d at io n would be t o main t ain a wide approac h in order t o
gen erate breadt h of c overage. Dec ision s about how t o set parame t e rs on large n u mbe rs
of bibliographic referen c es c an be made on c e some sen se of t he vol ume an d gen eral
sc ope of t he field has been gain ed.
Framework Stage 2: Identifying Relevant Studies
As already in dic ated, the whole poin t of sc opin g t he field is t o be as c omprehen sive as
possible in iden tifyin g primary studies (publ ished an d un publ ishe d) an d reviews suit -
able for an swerin g t he c en tral researc h quest ion . T o ac hieve t his, we adopt e d a strategy
t hat in volved searc hin g for researc h eviden c e via differen t sourc es:
elec tron ic databases
referen c e lists
han d-searc hin g of key journ al s
existin g n et works, relevan t organ izat ion s an d c on feren c es
From a prac tic al po in t of view, dec ision s have t o be made at t he out set abo u t t he c over-
age of t he review in t erms of t ime span an d lan guage. Reflec tin g t ime an d budge t
c on st rain t s, we in c l uded on ly t hose studies publ ished between Jan uary 1985 an d Oc t o -
ber 2001. T he st art dat e of 1985 was c hosen bec ause it was felt t hat t his c overed major
polic y c han ges in t he UK an d bec ause su ppo rt for c arers is relatively rec en t. Foreign
24 H. Arksey & L. O'Malley
lan guage material was exc luded bec ause of the c ost an d t ime in volved in t ran sl at in g
material. Whilst we had t o adopt these limits for prac tic al reason s, it is wort h poin t in g
out t hat poten tially relevan t papers c ould have been missed.
Electronic databases, the Internet and research registers
Elec tron ic databases usually c on t ain bibl iographic details an d abstrac ts of published
material. T here are a n u mbe r of issues researc hers n eed to c on sider before un de rt akin g
this impo rt an t stage of the proc ess suc h as: whic h databases t o searc h; what kin ds of
related t erms might be appropriat e to searc h for, in addit ion to key c on c ept s; pilotin g
the searc h strategy t o allow for refin emen t; whet her an y tec hn ic al searc hin g skills are
available to assist with t he searc hes; an d what the pot en t ial c osts are of on l in e ac c ess t o
elec tron ic databases, in t er-l ibrary loan s an d phot oc opyin g full artic les t hat are available
loc ally.
T he searc h strategy for elec tron ic databases is developed from the researc h question
an d defin ition s of key c on c ept s. Researc hers may n ot have t he skills n ec essary for
design in g an d exec utin g sen sitive searc h strategies t hat qualified l ibrarian s have. An
In format ion Offic er' from CRD worked with us to iden tify the relevan t keywords
(whic h may differ from on e database t o an o t he r); she also advised on what databases
were most likely t o produc e t he type of studies we were seekin g. She t hen devised an
in itial searc h strategy, whic h was later refin ed in the light of early results. T he fin al
version was first used on the MEDLINE database an d t hen c on verted for eac h subse-
que n t dat abase.
For o u r study, searc hes were made on 12 databases available from CD-ROMs, an d
four via t he In t ern et . T here were huge variat ion s in the n u mbe r of referen c es gen erated
by eac h database with four databases produc in g less t han 10 hits eac h an d two
(MEDLINE an d EMBASE) produc in g 1,565 an d 1,589, respec tively. It is n ot kn own
how effec tive the differen t databases were in gen erat in g the 204 artic les t hat were even -
tually in c luded in the fin al selec tion . Suc h in format ion c ould be useful for an y similar
work likely to be un de rt ake n in t he future.
Reference lists
We foun d it valuable t o c hec k the bibliographies of studies foun d t hrough t he database
searc hesespec ially systematic reviews an d t radit ion al l it erat ure reviewsto en sure
they had been in c l uded in t he sc opin g exerc ise. T his proc ess did iden tify further refer-
en c es, al t hough a sat urat ion poin t was reac hed where n o n ew on es were bein g iden t i-
fied. Citation searc hes might have also yielded n ew st udies, al t hough we did n o t utilize
this t ec hn ique.
Hand-searching of key journal
It is impo rt an t t hat key journ al s are han d-searc hed t o iden tify artic les t hat have been
missed in database an d referen c e list searc hes. T his c an oc c ur bec ause elec tron ic dat a-
bases may be in c ompl et e, n ot up t o dat e or bec ause abst rac t in g servic es c an vary in
International Journal of Social Research Methodology 25
c overage, in dexin g an d dept h of in format ion . Al t hough most databases c on t ain a
pro po rt io n of British journ al s, they all t en d t o have a West ern an d partic ularly US bias.
We iden tified four c o mmo n journ al titles t hat we felt required han d-searc hin g.
Un fort un at el y, n ot all of t he journ al s were available at the Un iversity of York whic h
mean t travellin g some distan c e t o the n earest library t hat subsc ribed to t he journ al (s)
in quest ion , an un an t ic ipat ed ac tivity t hat added t o t he pressure on t ime an d resourc es.
Existing networks, relevant organizations and conferences
As ot her researc hers un de rt akin g reviews have foun d (Badger, Nurst e n , Williams, &
Woodward, 2000), usin g existin g kn owledge an d n et works c an gen erate in format ion
about primary researc h. So, t o o , c an c on t ac t in g relevan t n at ion al or loc al organ izat ion s
workin g in t he field, with a view t o han d-searc hin g libraries an d /o r iden tifyin g u n pu b-
lished work. We c on t ac t ed a n u mbe r of relevan t organ izat ion s in c l udin g Carers UK,
the Sain sbury Cen t re for Men t al Heal t h, the Men t al Health Fo u n d at io n , the Kin g's
Fun d, an d the Nat ion al Sc hizophren ia Fellowship.
T he searc h c an gen erate man y t ho u san d s of bibl iographic referen c es whic h t hen
n eed appraisin g to see whet her or n ot they shoul d be in c l uded in t he fin al st udy selec -
t ion . Bibliographic software pac kages suc h as Referen c e Man ager or En dn ot e , an d
gen eral text retrieval databases suc h as Idealist, are useful dat a man age me n t t ool s. We
used an En dn ot e library whic h proved in valuable for man agin g rec ords, keepin g trac k
of artic les an d makin g requests for in t er-l ibrary l oan s. T he En d n o t e software was
c ompat ibl e with the word proc essin g pac kage we were usin g, an d it was a relatively
quic k an d easy task t o pro d u c e lists of referen c es for in c lusion in t he fin al l it erat ure
review report . T he In format ion Offic er rec orded eac h dat abase searc hed, t he years it
c overed, an d t he date it was searc hed for eac h set of results when they were import e d
in t o En dn ot e . Kn owin g what databases were searc hed an d from what dat e is impo rt an t ,
espec ially if t here is an y likelihood of havin g t o updat e t he searc hes in the future.
T he various mec han isms for searc hin g in o u r sc opin g st udy gen erated a total of
3, 867 referen c es, some 112 of whic h were iden tified as t he st udy progressed (these were
t reat ed in the same way as t hose gen erated in t he main elec tron ic bibl iographic dat a-
base searc h). T he majority of referen c es (3, 755) were foun d on t he elec tron ic bibl io-
graphic databases, whic h further emphasizes t he import an c e of developin g skills in this
area.
Framework Stage 3: Study Selection
Ou r in itial perusal of the c it at ion s in dic at ed t hat t he searc h strategy had pic ked up a
large n u mbe r of irrelevan t st udies. T his hn ks t o t he impo rt an c e of defin in g t e rmin ol -
ogy at t he outset of a sc opin g study, an d in o u r c ase refiec ts some spec ific diffic ulties
suc h as differen t c oun t ry's t ermin ol ogy t o desc ribe c arers, an d t he fac t t hat we had
sought breadt h rat her t han de pt h.
We n eeded a mec han ism t o help us el imin at e studies t hat did n ot address o u r c en tral
researc h quest ion . Systematic review me t hods develop in c lusion an d exc lusion c riteria,
based on a spec ific researc h quest ion , at the out set of t he projec t t o en sure c on sisten c y
26 H. Arksey & L O'Malley
in dec ision -makin g. Ou r sc opin g st udy adopt ed similar me t hods, al t hough c riteria
were devised post hoc, based on in c reasin g familiarity with t he l it erat ure, t hat we c ould
t hen apply t o all t he c it at ion s t o de t e rmin e t heir relevan c e. T he in c lusion c riteria used
in o u r sc opin g st udy related t o t he: type of study; type of in t erven t ion ; c are rec ipien t
group; an d c arer g ro u p.
T wo reviewers t hen applied t he in c lusion an d exc lusion c riteria t o all the c it at ion s.
Copies of t he full artic le were obt ain ed for t hose studies t hat appeared t o represen t a
'best fit' with t he researc h quest ion . If t he relevan c e of a st udy was un c lear from t he
abstrac t, t hen t he full artic le was ordered. A deadlin e was set, after whic h it was agreed
t hat we would n o t in c l ude an y more studies in t he an alysis. T his is an impo rt an t dec i-
sion t o make when t ime is limited, al t hough it is good prac tic e t o in dic at e in an appe n -
dix an y artic les t hat have n ot been reviewed but whic h may be of in t erest t o ot her
researc hers. T he n ext stage requires reviewers t o read t he full artic les t o make the fin al
dec ision about whet her they shoul d be c hosen for in c lusion in t he review. As Badger et
al. (2000) n ot e , abstrac ts c an n ot be assumed to be represen tative of t he full artic le t hat
follows, or t o c apt ure the full sc ope of an artic le.
Ou t of o u r origin al 3, 867 referen c es, 453 were ordered t hro u g h in t er-l ibrary loan s;
some 30 or so were available loc ally for phot oc opyin g. Havin g read the artic les in full,
204 artic les were selec ted for in c lusion in t he review.
Framework Stage 4: Charting the Data
T he n ext stage of t he work in volved 'c hart in g ' key items of in format ion obt ain ed from
the primary researc h report s bein g reviewed. 'Chart in g ' (Ritc hie an d Spen c er, 1994)
desc ribes a t e c hn ique for syn thesizin g an d in t erpret in g qualitative data by siftin g,
c hart in g an d sort in g mat erial ac c ordin g t o key issues an d t hemes, a similar proc ess t o
the on e we adopt ed hen c e we have borrowe d t he t e rm. In a systematic review, this pro c -
ess would be c alled 'dat a ext rac t ion ' an d, in t he c ase of meta-an alysis, might in volve
spec ific statistic al t ec hn iques.
Ou r c hart in g approac h was akin t o a 'n arrat ive review' (Pawson , 2002, p. 171),
whic h takes a broade r view t hat c an in c l ude, for example, rec ordin g in format ion about
t he 'proc ess' of eac h pro g ramme or in t erven t ion in c l uded in t he review so t hat its
'o u t c o me ' is c on textualized an d more un de rst an dabl e t o readers. Dec ision s have t o be
taken abo u t what in format ion shoul d be rec orded from t he primary st udies, an d it is
impo rt an t t o c on sider how c omparison s between differen t in t erven t ion s c an be
ac hieved. Simply produc in g a short su mmary or profile of eac h st udy does n o t guaran -
tee hel pin g t hose readers who might have t o make impo rt an t dec ision s based on t he
st udy fin din gs (Pawson , 2002). T he 'desc ript ive-an al yt ic al ' me t ho d wit hin t he n arra-
tive t radit ion , whic h in volves applyin g a c o mmo n an alytic al framework t o all t he
primary researc h report s an d c ollec tin g st an dard in format ion on eac h study, st an ds
mo re c han c e of bein g useful.
T he data t hat we c hart ed were en t ered o n t o a 'dat a c hart in g form' usin g t he database
pro g ramme Exc el. What shoul d t he c on t e n t of dat a c hart in g forms in c lude? Cen erally
speakin g, this will be a mixt ure of gen eral in format ion about t he st udy an d spec ific
International Journal of Social Research Methodology 27
in format ion relatin g t o, for in stan c e, t he st udy po pu l at io n , t he type of in t erven t ion ,
out c ome measures employed an d the st udy design . We rec orded in format ion as
follows:
Au t ho r(s), year of publ ic at ion , st udy loc ation
In t erven t ion type, an d c omparat or (if an y); d u rat io n of the in t erven t ion
Study popul at ion s (c arer group; c are rec ipien t group)
Aims of t he st udy
Met hodol ogy
Ou t c o me measures
Impo rt an t results
Addit ion al st an dardized data were ext rapol at ed from t hose studies with an ec on omic
c o mpo n e n t . T ogether, these data formed t he basis of t he an alysis.
We sought a un iform approac h t o all 204 studies in c l uded in the review, al t hough in
prac tic e it was often impossible t o extrac t all t he in format ion required where researc h
report s failed t o in c l ude relevan t mat erial . As ot hers have n ot e d (Badger et al. , 2000),
data are n ot always presen t ed in t he most ac c essible of formats.
Framework Stage 5: Collating, Summarizing and Reporting the Results
T his stage of a sc opin g st udy in volves c ollatin g, summarizin g an d re port in g the results.
Again , we c an make useful c omparison s between the sc opin g st udy an d t he full system-
atic review. Whilst the proc ess of c ollec tin g an d reviewin g studies for a full systematic
review may require researc hers t o read an d review a large n u mbe r of st udies, on l y a
small perc en tage may be in c l uded in the fin al re port . Eviden c e or fin din gs from st udies
n o t in c l uded in the fin al review may c on sequen t l y remain hidde n from publ ic at ion . In
c on t rast , t he sc opin g st udy seeks t o presen t an overview of all mat erial reviewed an d
c on sequen t l y issues of how best t o presen t t his pot en t ial l y large body of mat erial are
c ritic al.
Moreover, un like a systematic review t he sc opin g st udy does n ot seek t o 'syn t hesize'
eviden c e or t o aggregate fin din gs from differen t st udies. Whil st a sc opin g st udy will
n eed some an alytic framework or t hemat ic c on st ruc t ion in order t o presen t a n arrat ive
ac c oun t of existin g l it erat ure, t here is n o at t e mpt made t o presen t a view regardin g the
'weight ' of eviden c e in relation t o part ic ul ar in t erven t ion s or polic ies. T his is bec ause
t he sc opin g st udy does n ot seek t o assess quality of eviden c e an d c on sequen t l y c an n o t
det ermin e whet her part ic ul ar studies provide robust or gen eralizable fin din gs.
Havin g 'c hart e d' in format ion from studies, we were able t o presen t o u r n arrat ive
ac c oun t of fin din gs in two ways. Eirst, at t e n t ion was given t o basic n umeric al an alysis
of t he exten t, n at u re an d dist ribut ion of t he studies in c l uded in t he review. We
produc e d tables an d c hart s mappin g: t he dist ribut ion of studies geographic ally an d for
t he differen t c are rec ipien t groups; t he ran ge of in t erven t ion s in c l uded in t he review;
the researc h me t hods adopt ed an d t he measures of effec tiven ess used. T his part of t he
an alysis shed light on t he d o min an t areas of researc h in t erms of in t erven t ion t ype.
28 H. Arksey & L. O'Malley
researc h me t hods an d geographic al l oc at ion . We c oul d very quic kly get a flavour of the
main areas of in terest, an d c on sequen t l y where t he sign ific an t gaps were.
Sec on d, t he l it erat ure was organ ized thematic ally ac c ordin g to 11 differen t in t erven -
t ion types. T his was an ot he r diffic ult an d t ime -c o n su min g ac tivity sin c e t here was great
diversity an d /o r overlaps amo n g report s; desc ript ion s of some in t erven t ion s were
in suffic ien t; an d au t ho rs' defin ition s did n ot always appear justifiable or c on sist en t .
T he in t erven t ion type bec ame t he primary un it of an alysis an d o u r fin al l it erat ure
review report was organ ized aro u n d these 11 c ategories (see Arksey, O'Malley,
Baldwin , & Harris, 2002).
In developin g a framework for c ollatin g an d summarizin g results, the sc opin g st udy
does forc e researc hers t o prioritize c ertain aspec ts of t he l it erat ure. By adopt in g an
approac h based on in t erven t ion type, o u r fin din gs t en ded t o subsume theoretic al or
c on c ept ual posit ion s adopt e d by au t ho rs. An altern ative approac h may have been to
base o u r an alysis on c ompet in g t heories of c arer in t erven t ion s (suc h as 'family t herapy'
or 'c ogn itive behavioural t herapy). T o this exten t it is c ruc ial t hat t he sc opin g st udy
me t ho d retain s a c larity of report in g strategy so t hat t he reader c an de t e rmin e an y
pot en t ial bias in report in g or re c o mme n d at io n s. As with an y good quality researc h, t he
posit ion , or pot en t ial bias, of an y work must be iden tified an d pot en t ial l y subjec tive
dec ision s regardin g data an alysis made c lear.
Wit h t his in min d, we sought t o provide a c on sisten t approac h t o report in g o u r fin d-
in gs an d developed a 't e mpl at e ' t hat we applied t o eac h in t erven t ion g ro u p. T he
t empl at e began with a small table summarizin g basic c harac teristic s of all the studies
in c luded in t hat part ic ul ar in t erven t ion group, an d was followed by c o mme n t ary writ-
ten un de r t he followin g n in e headin gs: in t erven t ion s; sample sizes; part ic ipan t s;
researc h me t hods; out c omes; eviden c e relatin g t o effec tiven ess; ec on omic aspec ts; UK
studies; gaps in the researc h.
By applyin g a c on sist en t approac h t o re port in g t he fin din gs we were able t o make
c omparison s ac ross in t erven t ion types; iden tify c on t radic t ory eviden c e regardin g
spec ific in t erven t ion s; iden tify gaps in t he eviden c e base about in dividual in t erven t ion s
an d ac ross in t erven t ion s as well as c on sider possible 'n ew fron tiers' (suc h as t he In t e r-
n e t ). Of itself, the l it erat ure review (Arksey et al. , 2002) provided a c omprehen sive an d
t horough review of available l it erat ure an d iden tified n u me ro u s gaps in t he eviden c e
base.
T he iden tific ation of researc h gaps in o u r st udy relied on two main sourc es: the liter-
at ure review, whic h was c on fin ed t o iden tifyin g areas of overall weakn ess wit hin t he
field by c omparin g ac ross in t erven t ion types an d st udy design s; an d t he c on sul t at ion
exerc ise whic h proved in valuable for iden tifyin g c urre n t issues fac in g prac t it ion ers an d
c arers themselves t hat remain ed un der-researc hed. It is t o this fin al, an d opt ion al , stage
of the framework t hat we n ow t u rn .
Framework Optional Stage: Consultation Exercise
Eviden c e (Oliver, 2001) suggests t hat systematic reviews c an be en han c ed, an d t he
results made more useful, if prac t it ion ers an d c on sume rs c o n t ribu t e t o t he work
International Journal of Social Research Methodology 29
(Oliver, 2001). In deed, t here n ow exists a Coc hran e Col l aborat ion Co n su me r Net work
t hat in c ludes in dividuals an d c o mmu n it y organ izat ion s worldwide. T he Net work
support s an d develops c on sumer part ic ipat ion in the Col l aborat ion , an d helps make
t he in format ion available to c on sumers.
In t he light of o u r experien c es of t he sc opin g study, we would c ertain ly en dorse this
approac h. As in dic at ed at t he st art , in addit ion t o t he l it erat ure review, t he sc opin g
st udy also in c luded a c on sul t at ion elemen t (see Ne wbro n n e r & Hare, 2002). T his
in volved t hree groups of st akehol ders: represen tatives from n at ion al st at ut ory an d
vol un t ary bodies; man agers an d prac t it ion ers from loc al organ izat ion s; an d 'key
in forman t ' c arers.
Co n t ribu t o rs t o t he c on sul t at ion provided addit ion al referen c es about pot en t ial
studies t o in c l ude in t he review as well as valuable in sights about issues relatin g to the
effec tiven ess an d c ost-effec tiven ess of servic es t hat t he sc opin g review al on e would n ot
have alerted us t o. For in st an c e, whereas primary researc h report s t en ded t o foc us on
c han ges in levels of c arer bu rd e n , stress or kn owledge as a measure of effec tiven ess,
c o n t ribu t o rs approac hed t he c on c ept in a more ro u n d e d an d holistic way t hat e n c o m-
passed five related dimen sion s: ben efits for the c arer; ben efits for t he c are rec ipien t;
ben efits for t he family as a whole; t he impac t of servic e usage; an d, lon g t erm out c omes
for soc iety. T his perspec tive pro mpt e d the researc h t eam to quest ion t he pre d o min an c e
of t he use of st an dard out c ome measures. When report in g t he fin din gs, we suggested
t hat altern ative approac hes t o det ermin in g t he effec tiven ess of in t erven t ion s alon gside
st an dard out c ome measures shoul d be developed an d applied. Although t his el emen t
of o u r approac h t o a sc opin g st udy may be c on sidered an 'opt ion al ext ra', t he c on sul -
t at ion exerc ise did in deed provide 'added value' to the l it erat ure review.
Resource Implications
Al t hough sc opin g studies are often lin ked to 'rapid ' appraisal , it would be wron g to
assume t hat this me t ho d represen t s either a 'quic k' or 'c he ap' o pt io n . Ou r review
employed t hree full-time equivalen t staff me mbe rs for six mo n t hs as well as the servic es
of an in format ion offic er t o c on duc t l it erat ure searc hes. T he c ost impl ic at ion s for
retrievin g d o c u me n t s t hro u g h in t er-l ibrary loan s an d t he t ime impl ic at ion s t hat go
alon g with this retrieval mean t hat the sc opin g st udy shoul d n ot be seen as a c heap
altern ative t o t he systematic review, an d c on sequen t l y we would urge fun ders an d
researc hers t o be c aut ious in assumin g t hat a sc opin g st udy has sign ific an tly fewer
resourc e impl ic at ion s t han a systematic review.
Discussion and Conclusion
T he sc opin g st udy framework we have presen t ed in this paper c omprises five stages,
t oget her with an opt ion al c on sul t at ion exerc ise. It is based on o u r experien c es of, an d
l earn in g from, un dert akin g suc h a study. As we said at t he out set , t here is n o defin itive
proc e dure for sc opin g t he l it erat ure, an d we are n ot suggestin g t hat t he framework
presen t ed above is the on ly 'rig ht ' met hodol ogic al approac h t o take. On reflec tion , an d
30 H. Arksey & L. O'Malley
in t he light of c omme n t s from c olleagues in CRD, it is probabl y fair t o say t hat o u r
model of c on duc t in g a sc opin g st udy shared a n u mbe r of proc esses assoc iated with
systematic reviews.
T he proposed framework in c ludes a role for key stakeholder groups, in t he belief
t hat in c l udin g t he perspec tives of ot hers with kn owledge of, an d a vested in t erest in , t he
area u n d e r examin at ion gives an impo rt an t addit ion al dime n sion t o t he reviewin g
proc ess. T he framework also reflec ts the import an c e of tec hn ologic al devel opmen t s
an d expertise required t o retrieve an d man age dat a. T o t hat ext en t , sc opin g st udy me t h-
ods may represen t a shift in met hodol ogic al foc us away from expert kn owledge of a
part ic ul ar field assoc iated with the t radit ion al l it erat ure review, t owards an approac h
t hat emphasizes skills assoc iated with tec hn ic al kn owledge.
A key st ren gt h of t he sc opin g st udy is t hat it c an provide a rigorous an d t ran spare n t
me t ho d for mappin g areas of researc h. In a relatively short spac e of t ime (c ompared
with full systematic review), reviewers are in a posit ion t o illustrate t he field of in terest
in t erms of t he vol ume, n at ure an d c harac teristic s of t he primary researc h. T his an alysis
in t u rn makes it possible t o iden tify t he gaps in the eviden c e base, as well as su mmariz-
in g an d dissemin at in g researc h fin din gs. By presen t in g t he results in an ac c essible an d
summarized format, polic y makers, prac t it ion ers an d c on sume rs are bet t er plac ed t o
make effec tive use of t he fin din gs.
It woul d be misleadin g of us n ot t o ac kn owledge t he l imit at ion s of sc opin g st udies.
T hey do n ot , for example, appraise t he quality of eviden c e in t he primary researc h
report s in an y formal sen se. T he quan t it y of data gen erated c an be c on siderabl e. T his
c an lead to diffic ult dec ision s about how far breadt h (c overin g all available mat erial ) is
more impo rt an t t han dept h (providin g a detailed an alysis an d appraisal of a smaller
n u mbe r of st udies). T he sc opin g st udy does n ot address t he issue o f'syn t he sis', t hat is
the relative weight of eviden c e in favour of the effec tiven ess of an y part ic ul ar in t erven -
t ion . Con sequen t l y, sc opin g studies provide a n arrat ive or desc riptive ac c oun t of avail-
able researc h. Man y of these diffic ulties are addressed by systematic review me t ho d s
t hat do require quality appraisal, t hereby (mostly) reduc in g t he quan t it y of studies
in c l uded in t he review an d plac in g an emphasis on syn thesizin g dat a. However, t he
systematic review proc ess c an be very len gthy, a key disadvan tage when polic y makers
wan t in format ion about existin g researc h eviden c e soon er rat her t han later.
It would be wron g t o view t he sc opin g st udy me t ho d as an easy opt ion simply
bec ause hard quest ion s about quality appraisal an d syn thesis are avoided. Co n d u c t in g
a sc opin g st udy requires reviewers t o have high degrees of an alytic skill in order t o
develop frameworks t hro u g h whic h large n u mbe rs of studies c an be desc ribed.
Eurt he rmore , by n o t addressin g issues of quality appraisal, t he sc opin g st udy po t e n -
tially has t o deal with a greater ran ge of st udy design s an d met hodol ogies t han t he
systematic review, whic h has t en ded t o foc us on the ran domized c on t rol trial as the
gold st an dard of researc h design (CRD, 2001). Al t hough efforts are bein g mad e t o
develop t ec hn iques for t he appraisal an d syn thesis of qualitative data wit hin t he
systematic review c o mmu n it y (see, for example, Dixon -Woods, Eitzpatric k, &
Robert s, 2001), it remain s t he c ase t hat t he sc opin g st udy is mo re likely t o in c l ude an d
dissemin at e fin din gs from a ran ge of differen t me t ho d s an d st udy design s. Yet at t he
International Journal of Social Research Methodology 31
same t ime t he sc opin g st udy does n o t offer an y c lear mean s of syn thesizin g fin din gs
from differen t kin ds of st udy design . T hese issues require further at t e n t ion if sc opin g
studies are t o develop an d have a future in advan c in g t he eviden c e base in health an d
soc ial c are.
On e of the purposes of t he presen t paper is to st imul at e disc ussion about t he merit s
of sc opin g studies, an d help develop appropriat e me t ho d s for c on duc t in g suc h
reviews. An addit ion al aim for this paper is t o provide t he st art in g po in t for a wider
debate about t he role of the sc opin g st udy in relation t o ot her types of l it erat ure
reviews: where does on e en d an d the ot her start? We look forward t o seein g how the
debate progresses.
Acknowledgements
T his paper is based on a st udy fun ded by a gran t from t he UK NHS Servic e Delivery
& Organ izat ion (SDO) Researc h an d Devel opmen t Pro g ramme . T he views expressed
are t hose of the aut hors an d n ot n ec essarily shared by an y in dividual , gove rn me n t or
agen c y. We wish t o t han k o u r c olleagues on this projec t: Sally Baldwin (De part me n t
of Soc ial Polic y an d Soc ial Work, Un iversity of York) an d Jen n ifer Harris (SPRU,
Un iversity of York); An n e Mason (Cen t re for Heal t h Ec on omic s, Un iversity of
York); Su Golder (NHS Cen t re for Reviews an d Dissemin at ion , Un iversity of York);
Elizabeth Ne wbron n e r an d Philippa Hare (Ac t on . Shapiro). Sally Baldwin died tragi-
c ally in Oc t ober 2003. Sally was a key me mbe r of t he researc h t eam, an d we greatly
valued her in pu t t o t he projec t an d t his artic le. We are grateful t o Julie Glan ville,
Su Golder, Rac hel Pet o, Gill Ritc hie, Mark Rodgers, Aman da Sowden , Alison
Wallac e an d Kath Wright for c o mme n t s on an earlier draft of this paper. We would
also like t o ac kn owledge the referees' c o mme n t s whic h have been in c orporat e d in t o
t he paper.
Note
[1] We would like to ac kn owledge Su Golder's valuable support an d c on tribution to the sc opin g
study.
References
An tman , E., Lau, J., Kupein ic k, B., Mosteller, F. , & Chalmers, T . (1992). A c omparison of results of
meta-an alysis of RCT s an d rec ommen dation s of c lin ic al experts. Journal of American Medical
Association, 268, 240-248.
Arksey, H. , O'Malley, L., Baldwin , S., & Harris, J. (2002). Services to support carers of people with
mental health problems: Literature review report. York: Soc ial Polic y Researc h Un it, Un iversity
of York; available on lin e at: www. sdo. lshtm. ac . uk/men talhealthc arers. htm.
Badger, D. , Nursten , J., Williams, P. , & Woodward, M. (2000). Should all literature reviews be
systematic ? Evaluation and Research in Education, 14, 220-230.
Cen tre for Reviews an d Dissemin ation (CRD) (2001). Undertaking systematic reviews of research on
effectiveness: CRD's guidance for those carrying out or commissioning reviews, CRD Report 4
(2n d ed. ). York: NHS Cen tre for Reviews an d Dissemin ation , Un iversity of York.
32 H. Arksey & L O'Malley
Coc hran e Collaboration on Effec tive Profession al Prac tic e Un it (CCEPP) (1996). The data collection
checklist. York: Coc hran e Collaboration on Effec tive Profession al Prac tic e Un it.
Dixon -Woods, M. , Fitzpatric k, R., & Roberts, K. (2001). In c ludin g qualitative researc h in systematic
reviews: opportun ities an d prob\ems. lournal of Evaluation in Glinical Practice, 7, 125-133.
Hagell, A., & Bourke Dowlin g, S. (1999). Scoping review of literature on the health and care of mentally
disordered offenders, CRD Report 16. York: NHS Cen tre for Reviews an d Dissemin ation ,
Un iversity of York.
Jepson , R., Blasi, Z. D. , Wright, K., & Riet, G. T . (2001). Scoping review of the ejfectiveness of mental
health services, CRD Report 21. York: NHS Cen tre for Reviews an d Dissemin ation , Un iversity
of York.
Mays, N. , Roberts, E., & Popay, J. (2001). Syn thesisin g researc h eviden c e. In N. Fulop, P. Allen , A.
Clarke, & N. Blac k (Eds. ), Studying the organisation and delivery of health services: Research
methods. Lon don : Routiedge.
Newbron n er, E., & Hare, P. (2002). Services to support carers of people with mental health problems:
Gonsultation report. York: Soc ial Polic y Researc h Un it, Un iversity of York; available on lin e at:
vkfww. sdo. lshtm. ac . uk/men talhealthc arers. htm.
Oliver, S. (2001). Markin g researc h more useful: In tegratin g differen t perspec tives an d differen t
methods. In S. Oliver & G. Peersman (Eds. ), Using research for effective health promotion.
Buc kin gham: Open Un iversity Press.
Pawson , R. (2002). Eviden c e-based polic y: in searc h of a method. Evaluation, 8, 157-181.
Ritc hie, J., & Spen c er, L. (1994). Qualitative data an alysis for applied polic y researc h. In A. Bryman
& R. G. Burgess (Eds. ), Analysing qualitative data. Lon don : Routiedge.