BeyondtheMean DataanalysisforSchoolLeaders

GlenGilchrist AlexavierFareheed

ThiseditionpublishedbyLULU,February2012 ISBN:9781471611469 This work is licensed under a Creative Commons AttributionNonCommercial ShareAlike3.0UnportedLicense(CCBYNCSA3.0). Toviewacopyofthislicense,visithttp://creativecommons.org/licenses/byncsa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California94105,USA.WhilsttheCreativeCommonsLicenseforthisbookentitlesyou to distribute / modify the work for noncommercial use, without additional permissions, we kindly request that you inform the authors of any intention to re publish/remixthistitle.Sendanemailtomean@goingbeyond.co.uk Every effort has been made to contact perceived copyright holders for material reproduced in this publication. Any omissions or oversights will be rectified in subsequent editions if written notice is given to the author. All trademarks are the propertyoftheirrespectiveowners.Theauthorsarenotassociatedwithanyproduct or vendor mentioned in this book except where stated. Unless otherwise stated; any thirdparty quotes, images and screenshots, or portions thereof, are included under fairuseforcomment,newsreporting,teaching,scholarship,andresearch. Acknowledgements The authors would like to thank Michelle Gilchrist for her help, support are tireless proofreadingskills,withoutwhich,thisbookwouldnothaveseenthelightofday. Disclaimer Thisisabookaimedatthosereaderswantingtoexploredataasusedtodrivedecisions inschools.Itisnotacomprehensiveguidetostatisticsnoresponsibilityisassumed or accepted for your decisions based on your data. Using the techniques detailed in thistextprovidesanaidtodecisionmakingwhereas,thedecisiontoactislefttothe discretionofthereader.Noliabilitycanbeplacedwiththeauthorsofthistext. Byusingthematerialcontainedwithinthisguide,youacknowledgethatyouhaveread andacceptthisdisclaimer

Prefacetofirstedition
Ihavetoadmitalongstandingandgrowinginterestinthesubjectofstatistics.Asa researchscientistbeforefindingmyvocationasateacherIusedthetoolsofstatistics onadailybasistoinformmyresearchandtoplanfutureinvestigations.WhenIstarted onmyteachingcareerIwasamazedatjusthowunderdevelopedtheuseofproper numberswas,bothintheclassroomandwithinthewiderarenaofeducationalpolicy making.Farreachingdecisionsaremadeonthebasisofpoorlyresearchedandunder analyzed data. Everyones tax investments and our endeavors as a teacher/leader is constantly being misdirected by the improper analysis of data. This book is my contributiontothecauseofusingdatainanappropriateandconsideredmanner. Goodluckdearreader. GlenGilchrist,February2012 Ivebeenheadoffacultyfor5yearsnowandinallthattime,IdontthinkthatIveseen anyone literally anyone in the education sector use data in a robust manner. Sure, Ive seen pretty bar charts and tables used to justify interventions and to determine policy.IvesatthroughtoomanyINSETsessionsdiscussingtheconsequencesofpoorly analyzed data; in fact Ive been asked to lead on data sessions as presented to incomingPGCE,NQTsandnewstaffIguessinshort,Ivebecomepartoftheproblem. I believe that you dear reader have an obligation to reflect upon the data that you collectandtheconsequencesofyouranalysis. AlexavierFareheed,February2012 Correspondingwiththeauthors Data analysis can be a lonely pursuit. The authors are happy to receive questions, queriesandothercorrespondencesendanemailtomean@goingbeyond.co.uk.

Contents
Introduction Itseasytoseewhydataismishandledandunsafeconclusionsdrawn. Essentialdefinitions Awordaboutsoftware Minitab 13 Finalnote Chapter1 DATAANALYSISTHATSCHOOLSDO Whyweusethemeanaverage Factorswecancompare Centraltendency Themeanapointstatistic Moresophisticatedanalysis Complementingthemeanbarcharts Usingthemeantocomparesegmentsofdata Usingthelanguageofstatistics Thewiderschoolpicture Calltoaction Conclusions Chapter2 THEPROBLEMSWITHTHEMEAN Statisticsinaction Calltoaction Problemswiththemean Calltoaction Thedangersofpresumptionpreanalyzingthedata Calltoaction Whatdoyourbarchartsshow?
5

9 10 11 12 13 15 15 15 16 16 16 18 19 20 21 21 23 24 25 25 26 26 27 27 28 28 29

Ethics,politicsandgettingyourownway Calltoaction Howbiganeffect/differenceisbigenoughtomatter? Extrainformationinamodifiedbarchart CalltoAction Lookingatawholecohort Preconceptionsagain Conclusions Chapter3 COMPARATIVESTATISTICS Whatdoessignificantmean? Ttestsandpvalues CalculatingsignificanceusingExcel ExcelcommandforTtesting Calltoaction Conclusions Chapter4 FACTORSWITHMULTIPLELEVELS. Multilevelfactors Combinelevelstomakeabinarysolution Calculatingttestforbinneddata Limitsofthettest Multilevelfactors ANALYSISOFVARIANCE Doesattendanceaffectattainment? FittingatrendlinetoExceldata UsingR tocheckforgoodnessoffit OnewayAnalysisofVariance(ANOVA) Nonnumericmultilevelfactors Calltoaction Pauseforbreath.. Questionstoreflecton Conclusions
2

30 31 32 33 33 34 35 36 37 37 37 37 40 41 43 44 45 45 45 45 48 49 50 50 51 52 54 58 61 66 67 67 68

Chapter5 GENERALLINEARMODEL(GLM) ConstructingaGLM Deeperanalysis ExtendingtheGLM BuildinginteractionsintotheGLM BigimplicationsoftheGLM Calltoaction Conclusions Chapter6 MAINEFFECTS MainEffectsPlot InteractionsPlot Calltoaction Conclusions Chapter7 FINALREMARKS Toolsyoullneed:

69 69 70 73 75 77 79 80 80 81 81 83 84 88 88 89 89 89

Introduction
Every school leader, head of subject and class room teacher will recognize the followingscenario: Its a school INSET, and what wonderful pedagogical expertise is going to be shared withyou,thewillingstaff?Yes,youveguessitAddressingthegenderdifferential theverynamesendswavesofdjvuthroughthestaffandtheauthorsofthisbook developaninstantmigraine. Werenotdenyingthatthereisadifferencebetweenthegendersandtheirapproach toeducation;norarewesuggestingthatasteachersandleadersthatyoudontneed tomonitorthingstoensurethatsituationsarentimproving/deterioratingwhatbrings us to the point of tears, is that this statement is based on poorly and superficially analyzeddata. Aswewillshowinthisbook,itseasytoassumethatresponseswillbedifferentfora certainfactor,andwhenyoujustlookatthemeanofdataset,thisdifferenceisoften seen youve then proved your initial assumption and you dont look for a more fundamental root cause. In our experiences, this is the case with the gender differential,andIbetyouvefallenintoittoo. When we came into teaching, for the first time in our professional lives we became awareofthesituationofbeingdatarichbutinformationpoor.Educationabounds withnumbers,andschools,students&teachershaveneverbeenmeasuredasmuch astheyarein201020111 Butwhichnumbersdoyouuseandwhichdemandthatyoutakethemseriously?
1

WhilstthisappearstobeparticularlytrueoftheEnglish/Welshsystems,alleducationalinfrastructures

constantlybattlewithleaguetables,bandingandotherlists

 Itseasytoseewhydataismishandledandunsafeconclusionsdrawn. Until very recently, use of correct descriptive statistics was the preserve of the statistician,oftenresultinginthecalculationofarcanenumbers,utilizingimpenetrable mathematics.Indeed,pickupanythingbutthemostbasicofstatisticstextbooksand the reader will soon be swimming in a sea of mathematical notation, far beyond the readabilityofthosewithoutdegreesinmathematics. But with the change is responsibilities, the TLR structure, and the reduction is extraneousfunding,theexpectationisthatasasubject/schoolleader,youundertake dataanalysisanddrawconclusions. Idoubtyouretrainedinstatistics(andwhyshouldyoube?)soinsteadofcarryingout statistically valid analysis youve have returned to that most basic of measure the averageafterall,itseasytocalculateandmeanssomethingdoesntit? Throughout the text of this book, we will look at analysing the data a typical departmentinaschoolmightproduceinitiallybycalculatingmeansanddeveloping thisintoamorerigorousassessmentofdata. Sodearreader,thisbookisaimedatclassroompractitioners,headsofdepartmentand schoolleadersseekingadeeperunderstandingofwhatyourdataactuallyshows. Inanutshell,weregoingtotakeyoubeyondthemean. GlenGilchrist&AlexavierFareheed 2012

10

Essentialdefinitions
Weneedtodefinethreevitaltermsthatwillbeusedthroughoutthistext: Factor: Afactorisavariablewhosevaluesareindependentofchangesinthevaluesof Level: Factorscanbesplitintodifferentvalues.Statistically,thesevaluesarecalled 1=lessthan80% 0=80%to89.9% 1=90%andgreater Thenumericalvalueofthegroups(1,0,1)isnotimportantandthelabelsare usedtodentifythegroupedlevels.Someconsiderationneedstobemadeinto i
11

other variables. Traditionally factors are the groups into which we split our datagender,SEN,freeschoolmealsareexamplesofeducationalfactors.

levels. BinaryLevels Levels can be binary in nature boy or girl, SEN or not and can be representednumerically1=boy,2=girlorremainastext. MultilevelLevels Levelsarenotalwaysbinary,originatingprimaryschoolforexamplecouldbe one of 10 or more levels, with each school either referred to by name or a codednumber 1=SchoolA,2=SchoolBetc Levels can be numerical, quantitative or qualitative, binary or multi level.

For continuous levels (age and attendance are good examples) levels themselves might be grouped together to make analysis easier. These groupingsareoftencalledbinsandreferencewillbemadetobinsize. Attendanceforexamplecouldbebinnedas:

thesize/rangeofthegroupingsasthischoicecanaffectsubsequent data analysis however this is outside the scope of this text, and for theanalysisundertakeninschools,justensurethatthebinsare sensible.

Response:

Theresponseistheoutputthatyouaremeasuring.Forschoolbased data,averageortotalpointsscoreandnumberofCsarethetypical responsesmeasured.

Awordaboutsoftware MS Excel is referred to throughout this text and is used as convenient shorthand for spreadsheet. We acknowledge that other spread sheets such as OpenOffice and GoogleDocsareavailableandcanbeusedfairlyinterchangeablyforMSExcel(except where indicated). Each has their strengths / weaknesses, but all process statistical informationinmuchthesamemanner.Thereisnoneedtochangeyourspreadsheet packagetocompletethenumericalanalysisundertakeninthemajorityofthistext. Some of the more advanced statistics require the use of a dedicated statistics tool. Recently the cost of these tools has fallen dramatically and academic licenses can be obtained for less than 50. We cannot recommend strongly enough the value in obtainingthecorrecttooltoanalyzeyourdata. AgreatlistismaintainedatWikipedia,whichcomparesdifferentstatisticaltools,their costsandlicenses:http://en.wikipedia.org/wiki/Comparison_of_statistical_packages.

12

 Minitab ThroughoutthisbooktheauthorsmakesuseofMinitabasaconvenientlyeasytoolto get to grips with and available at an excellent price (from sub 20) (http://www.minitab.com/enGB/academic/licensingoptions.aspx).Thepublisheralso makesavailableafree30daytrialmorethanenoughtimetolearntheropesandto processdataforyourselfevaluation. Finalnote The authors are practicing teachers, currently heads of subject in maintained secondaryschoolsandhavenoassociationwithanyofthetools/software/publishers mentionedinthistext. Dataanalysisisajourneythattheonlydestinationisenlightenmentgetreadyfor therideofyourlife.Glen&AlexavierFebruary2012

13

14

Chapter1
Dataanalysisthatschoolsdo
Oneofthebiggestchallengesingettingdatausedcorrectlyinschoolsusedtobethe actualcollectionandmanualprocessingofthenumbers.NowwithtoolssuchasMS Excel, OpenOffice and GoogleDocs available to all, the challenge has shifted to the actualprocessingandanalysisthatturnsnumbersintodata. Courses abound in educational circles about the use of data, but from personal experiencestheyallfocuson3areas: 1. Sourcesofbaselinedata(CATs,FFT,Government,FeederPrimaries) 2. Segmentingthedata(gender,freeschoolmeals,SEN) 3. Monitoring,assessingandexplainingstudentperformanceagainst(1)and(2) Valuableasthesecoursesare(andasignificantimprovementonnotusingdata),they all focus on basic statistics the mean average, range and a cursory diversion into drawing and formatting bar / line graphs; and whilst this is encouraged, reliance on thesemeasuresalonecanleadtopoorlydrawnandcostlyconclusions. Whyweusethemeanaverage WhilstExceletalhavedemocratizedthecollectionandanalysisofdata,theyhavealso exposedthefactthatmostusersofthesetoolsareunawarehowtousethematahigh enoughleveltoprocessstatisticalinformation.Asaresult,mostusersarecontentwith tabulation, calculation of averages of data sets and with drawing basic, overly colouredbarcharts. These averages are then used to draw conclusions, usually in the form of comparisons; Boys vs Girls, free school meals vs non free school, English vs Maths, 2009vs2010,oneschoolvsanother.

15

Factorswecancompare Thecandidatelistforcomparisonislong:specialeducationalneeds,ethnicity,looked after, target group, literacy booster support or a hundredandone other educationalimperatives.AsituationthatIamcertainoccursinyourschool.Indeedthe schools inspection framework2 demands that schools use data to identify, plan and monitortheattainmentofgroupsoflearners.Withoutextensiveuseofsuchdata, schoolscannothopetoachieveacovetedGrade1status. Wewillexposeinthischapterthedangersofusingjustthemeantorepresentadata set, and show how drawing conclusions can lead to costly and unnecessary interventions. Centraltendency Usedinthiscontext,themeanisameasureofcentraltendency3 The two most widely used measures of "central tendency" of data are the mean (average)andthemedian.Forexample,tocalculatethemean weight of50people, addthe50weightstogetheranddivideby50.Tofindthemedian weight ofthe50 people,orderthedataandfindthenumberthatsplitsthedataintotwoequalparts. Themedianisgenerallyabettermeasureofthecentrewhenthereareextremevalues or outliers because it is not affected by the precise numerical values of the outliers themselves(Themedianisoftenusedtodescribeaverageearningsinapopulationas itisnotaffectedbyasmallnumberofverylarge(orsmall)salaries). Themeanapointstatistic Themeanisapointstatisticthatis,itreducesanentiredatasettoasinglevalue, usefultosuccinctlydescribethedata.(However,youloseanysenseofthespreadand variabilityofthenumbers).Asaresult,themeanisthemostwidelyusedmeasureof centraltendency,butaswewillsee,notalwaysthemostuseful.
2 3

UKwide,butcertainlyheavilyendorsedinEnglandandWales Therearethreemeasuresofcentraltendencyusedtodescribedatasetsmean,modeandmedian.If

youareunfamiliarwiththesetermsorjustneedarecap,rememberGoogleisyourfriend.

16

 Forexample,theAveragePointsscorefor5schoolsin2011was: School A B C D E Whatconclusionscanbedrawnfromthisdata? Its likely that such analysis is undertaken at this level in both your department and wholeschoolselfevaluations. Theconsequencesofsuchanalysisarelikelytobesomeformofchange,intervention orclosermonitoring.Inshort,money,timeandeffortwillbeexpendedactingonthis analysis of means. A situation that we are sure has happened in your school or department. SchoolCisthebestperforming SchoolBistheleastperforming SchoolsA,CandEallhavesimilarpointsscores SchoolBneedstodosomethingasitsperformanceisverydifferenttothe otherschools. Average PointsScore 435 403 440 427 438

17

 Moresophisticatedanalysis Further and seemingly more sophisticated analysis will have you looking at the same dataoveraperiodof3or5years: School A B C D E Whatdoesthisshow? As part of your self evaluation / action plan you will have undoubtedly looked at 3 yeartrendsinmeandata.Yourelikelytohavecomparedyourresultstothatofother departments,betweenlocal,nationalandfamilyofschoolsandmadepronouncements onhowwellyouaredoingcomparedtolastyear. Totryanunravelsomeofthemysteryaboutwhatyourdataisshowingyou,chances areyoulldrawabarchartofthemeans. SchoolCisthemostimprovedoverthe3years SchoolBhasfallen37pointsover3years SchoolsDandEhaveshownlittleimprovementoverthethreeyears 20082009 425 440 411 425 430 20092010 430 420 424 430 438 20102011 435 403 440 427 438

18

 Complementingthemeanbarcharts Letscompletetheanalysisanddrawabarchartofthedatafortheschoolsoverthree years:

Whatdoesthischartshowus? Overall,whatconclusionscanbedrawnaboutschoolsAtoE? SchoolAisdoingsomethingthatisimprovingperformance School C is clearly doing something better than the other schools and betterthanschoolA
19

ItemphasizesthefallinperformanceofschoolB TheperformancegainsofschoolClookincredible SchoolDlooksallbutstaticoverthepastthreeyears

SchoolDappearsnottobedoinganythingandperformanceisstatic School E looks like something happened during 20092010, but these gains havestoppedandtheschoolhasnotimprovedsince. SchoolBlookslikeitsinfreefallandstandardsarefallingrapidly

No doubt such analysis is regularly completed by you and/or your senior leadership team.Andifourpersonalexperiencesarereflectedinyourschoolthestresslevelsand anxietyrisesinproportiontothepreparationandanalysisofsuchdata. Usingthemeantocomparesegmentsofdata As a teacher, administrator or policy maker we often need to compare the means of two or more populations essentially to test whether or not an intervention or observationproducesameasurabledifference.Forexample,theaveragepointsscore forYear11studentsuponreceivingtheirL2qualificationsisoftensegmentedintodata formalesandfemales. Asaresultofthisbasicanalysis,decisionsandpolicywillbedecided. In this case, clearly there is a sex linked differential between Boys and Girls with Girls outperforming Boys by some 10%. From this analysis of means an intervention will be planned possibly grouping next years cohort into separate sex classes, planningboyfriendlylessonsandtweakingtheseatingplans. Again,weresurethatyourefamiliarwithsuchsegmentationofdataandarecertain that your self evaluation contains statements about the gender differential and how youintendtotackleit.
20

AveragePointsScore Boy Girl 402 448

Usingthelanguageofstatistics Atthispoint,letsstarttousethelanguageofstatisticsmorefully. Inthecaseaboveforboys/girlsL2performance: Wehaveonefactor,SEX,splitintotwolevels(BoyandGirl)wesaywehavea binaryfactor. Fromnowon,wewillusefactor,levelandresponsetodescribeourdata. Thewiderschoolpicture Suchanalysisisextendedacrossthewiderschool,comparingthedifferentialsinyour subjecttothoseinEnglish,FrenchandDT4asadirectresultofthisanalysisaworking party or even a PLC5 will be created to tackle the clear differences between subject areas. (Whilst written here in a tongueincheek manner, I suspect that your school has at some point created a working party to contemplate differences in responses when factorsareanalyzedformeandifferences) OurresponseistheAveragePointsScore

4 5

Insertthehighperformingsubjectareasinyourschool PLC Professional Learning Community, school based collaborative action research for more details

see:http://www.centerforcsri.org/plc/program.html

21

 Whatcanweconcludefromthischart? The temptation in this case is to view the French differential (low) as in some way betterthattheSciencedifferential(high)andtoinvesttimeandresourcesinsolving theproblem. Werenotsuggestingthatthisdoesnotneedtobesolved;justthatthedataanalysis performedsofardoesnotdemandsuchinvestigations,merelyhintsatit
22

Frenchhasthesmallestsexdifferential Sciencehasthewidestdifferential InDT,boysoutperformgirls

Calltoaction 1. Do you know the three measures of central tendency and when to use each one?DoyouknowhowtogetExceltocalculateeach? 2. Findyourselfevaluationandidentifywhereyouhaveusedthemeanofadata settodrawaconclusionaboutsegmentationofdata 3. Look at the charts and graphs you have created for your exam analysis meeting.Aretheybasedonmeansofdata?Whatconclusionsdidyoudraw fromthem? 4. Lookatwholeschool,localandnationaldatahowoftenisanentiredataset reducedtoapointstatistic? 5. Howwellcanyouuseyourspreadsheettools? a. Canyouenterformulatocalculatetheaverageofadataset? b. What about counting the numbers in a column when the value in a different column is a particular value? (CountIF() used to automaticallycountdata,saybasedonacolumncontainingthesexof alearner)

23

 Conclusions Duringthischapterwehaveshownthebasicdataanalysisundertakenbyschools.As subject team leader we imagine that you have laboured over such figures yourself, painstakinglyenteringfiguresintoMSExcel,creatingcomparisonbar/piechartsand drawingconclusionsbasedonthemeanaverageofdatasets. Youve likely taken such figures into exam analysis meetings with your head teacher anddrawnconclusionsaboutwhystudentswhoobtainfreeschoolmealsdolesswell inyoursubjectthan,say,Spanish. Allofthesethingsareastepintheroadtounderstandinghowtousedataeffectively andthefactthatyouarereadingthistitledemonstratesacleardesiretotakeyouruse ofdatatoahigher,moreeffectivelevel. InthecomingchaptersIllshowyouwhydataanalysisbasedsolelyonthemeanofa population is dangerously superficial and can lead to misdirected effort and the potentialtomissamorefundamentalunderlyingtruth.

24