INLIGHTOFTHELIMITATIONSOFDATA‐DRIVENDECISIONMAKING
SusannaLoeb
Students’experiencesandtheopportunitiestheyhavetolearnrestonthequalityof
educationdecisionsmadeineachclassroom,ineachschool,ineachdistrict,andineach
state,federallegislature,anddepartmentofeducation.Whocanrunschools?Whocan
teach?Whatcontentiscovered?Howisittaught?Whichstudentsareineachclass?When
isrecess?Howmuchmoneyisspent,onwhatandonwhom?Thesearejustsomeofthe
decisionsthatmustbemadeandwhoseanswersaffectstudents.Theroleofresearchand
scholarshipmorebroadlyineducationfinanceandpolicyistoinformthesedecisionsfor
thebenefitofstudents.Inpractice,however,theinfluenceofresearchisoftenopaque.The
lackoftransparencycomesnotonlyfrompoordecisionmakingbutalsofromgood
decisionmakersweighingincompleteinformationaboutthelikelyeffectsoftheiroptions
withalogicabouthowtheworldworks.Usefuleducationresearchbuildstheselogicsand
providesinformationabouttheeffectsofspecificchoices.Italsoidentifiesindividualswith
thegreatestpotentialtomakegooddecisionsgiventhisincompleteinformationandthe
needtousejudgmentindecisionmaking.
Data‐drivendecisionmakingisthemantraofmanyrecenteducationreformdiscussions.
TheWebsiteoftheInstituteofEducationSciences,theresearcharmoftheU.S.
DepartmentofEducation,states,“Byidentifyingwhatworks,whatdoesn’t,andwhy,we
aimtoimproveeducationaloutcomesforallstudents,particularlythoseatriskoffailure.”
TheCenterforDataDrivenReforminEducation(2011)“conductsresearchintohow
schooldistrictscanusedata‐drivenreformtoimprovestudentachievement.”Oneofthe
SpencerFoundation’s(2010)strategicinitiativesseekstounderstanddatauseand
educationalimprovement.Therhetoricclearlysupportsdatauseineducationdecision
making,buttheuseofdatainactualdecisionsisrarelydirectortransparent.
Moreinformationandmoreaccurateinformationcanimprovedecisionmaking,but
consequentialeducationdecisionsarealmostuniformlyandinherentlygray.Thatis,even
withthebestavailableinformationaboutthepresentandthepast,decisionsaboutthe
futurerelyonjudgmentsaswellasknowledge.Inchoosingamathcurricularpackage,for
example,itisusefultoconsulttheWhatWorksClearinghousetohelpidentifyprograms
thathavebeensuccessful;however,thereareunlikelytobeanyone‐size‐fits‐allsolutions.
Themeritsofaparticularcurriculumlikelydependonthelocalcontextandchangeasnew
programsandgoalsaredeveloped.
Areasonabledecisionmakerwillalsoconsidertheneedsofherstudentsandhowthose
needscomparewiththoseoftheparticipantsinthestudy.Shewillconsiderwhethermore
promisingapproacheshavebeendeveloped,evenifuntested,thatmightworkbettergiven
hergoalsandcontexts.Shewillconsiderthelikelyresponseofstudentstothedifferent
approaches,combininginformationaboutthestudents’needsandtheoptionsavailableto
meetthoseneedswithherownlogicsaboutfactorsthatmightbeaffectedbythechoices
shemakes.
Manychoiceshaveevenlesssupportiveresearchavailablethanthechoiceofcurricular
programs.Forexample,inconsideringwhichteachertohire,thereisevidencethata
master’sdegreedoesnotguaranteesuccessandthatteacherswithsomeexperienceare
moreeffectivethanthosewithnone.Thereisalsoevidencethatacandidate’sperformance
onavarietyofassessmentsmeasuringgeneralknowledge,knowledgeforteaching
mathematics,verbalability,andotherskillsanddispositionsarepredictiveoflatersuccess
intheclassroom.Butevenifdataonteachercharacteristicsthatarepredictiveofstudent
learningwereavailable,whichtheyrarelyare,aneffectivehiringauthorityknowsthe
limitationsinapplyingtheseaverageeffectstotheindividualsinfrontofherandwill
collectherowninformaldatafrominterviews,teachingdemonstrations,andlettersof
recommendation.Inmakingherchoice,shewillcombinesourcesofinformationwithher
ownlogicsofwhatskillsateacherneedstobesuccessfulinhisschool.
Themostwide‐rangingquestions—designingsystemsforschoolfinanceorgovernance,
choosingcertificationrequirements,orsettingupcharterschoollaws—areoftentheleast
directlyinformedbyempiricalresearch.Norandomizedexperimentshavetested(orin
manycasescantest)theefficacyofthesetypesofpolicies,andevencorrelational,quasi‐
experimentalstudiesarerareandofteninfeasible.Instead,decisionmakersdrawon
resourcessuchasdatathatareperipherallyrelatedtothedecisioninquestion,anecdotal
information,andtheirownlogicsaboutimportantmechanismstocon‐sider.Inchoosing
certificationrequirements,decisionmakersmaydrawoninformationaboutwhether
teacherswhohavetakenmoremathcoursesincollegearemoreeffectiveatincreasing
studentlearning.Whilerelevant,thisinformationcanalsobemisleading;evenifteachers
whohavetakenmorematharemoreeffective,mandatingadditionalcoursesina
certificationsys‐temmayfuelunintendedconsequences.Forexample,ifpoor‐quality,low‐
costcourseworkoptionsspringupifteachersseekcertificationbutarenotinterestedin
theirownimprovement.Alternatively,theobservedrelationshipbetweenmathclassesand
effectivenessmaybedrivenbymoreeffectiveteacherschoosingtotakemoremath,and
notthecausaleffectofmathclassesoneffectiveness.
Giventheinherentlyincompleteinformationavailableabouttheeffectsofeachoftheir
choices,decisionmakersdrawontheirlogicaboutthelikelyeffectsofeachchoice.Logic
mightwarndecisionmakersawayfromchoosingafinancesystemthatprovidesmore
moneytodistrictsthatcategorizemorechildrenasspecialeducationstudentsorfroma
charteringsystemthatas‐signsdistricts—potentialcompetitorstonewcharters—asthe
solecharteringauthority.Theymayworryabouttheresponsesoflocalactorsasthe
systemcreatesincentivesforthemtoclassifymorestudentsortocharterfewerandless
competitivecharters.
Manylogicscomefromindividualexperiences,suchasseeingastudenthelpedorhurtby
aninstructionalapproachoranavailablesupportservice,orseeingateachermovetoa
schoolwithahigher‐scoringstudentpopulationoracollegialworkenvironment.Butoften
logicscomefromsharedexperiencescapturedthroughresearchorscholarship—poorly
designedaccountabilitysys‐temscreatestrongincentivestogame,itiseasiertoattract
teacherstosomeschoolsoverothersevenwhensalariesandbenefitsarethesame,and
cur‐ricularmaterialsthatworkwellwithnativeEnglishspeakersmaynotworkwellwith
Englishlearners.Evenifresearchcannotidentifytheeffectsofeachchoiceadecision
makerhas(anditrarelycan),scholarshipcaninformchoicesbydevelopingandsolidifying
theselogics.Researchbothidentifiespotentialmechanismsandprovidesestimatesofthe
likelystrengthorsalienceofeachofthesemechanisms.Asanexample,inthinkingabout
specialeducationfinance,onemightidentifythepotentialresponseofdistrictstomonetary
incentivestoclassifymorestudentsasinneedofspecialeducation;empiricalresearchcan
theninformtheextenttowhichdistrictstendtorespondstronglyorweaklytothese
incentives.
Insomedecisions,lackofknowledge(orfacts)isabindingconstraintingooddecision
making,butanarrowfocusonincreasingthisknowledgecanobscuretheimportanceof
judgmentandboththeindividualexperiencesandthepublicscholarshipthatgeneratethis
judgment.Randomizedexperimentsthatevaluatecurriculaandprofessionaldevelopment
programsorresearchthatusesmultiplemeasuresofstudents’progressandteachers’
instructionalpracticeimproveourknowledgeofwhathasworked.However,most
decisionsarebasedonlyindirectlyonthesefacts.Instead,theycomefromthetheoriesand
frameworksforhowtheworldworksthatarisefromthesefactsaswellasfromeach
decisionmaker’sexperience.Thescholarshipthatcontributestotheseframeworks,by
synthesizingtheinternallyvalidexperimentsaswellasthedescriptionsoftheworld,are
moreproximatetomosteducationdecisionsthantheresultsofindividualresearch
projects.
Theabovediscussionhighlightsthreenoteworthyresourcesforeducationaldecision
making.First,decisionmakersneedinformationabouttheirneeds:Dostudentsneedhelp
withvocabularyorgrammar?Fractionsordecimals?Doestheschoolhavetroublefinding
teacherswithstrongcontentknowledgeorstronginstructionalskills?Second,good
decisionsrelyonsolidinformationabouttheoptions.Ifthedecisionmakerisnotawareof
thebestoptions,heorshewillnotchoosethem.Third,gooddecisionsdependonabroad
understandingofhowtheworldworks,particularlyregardingthedecisioninquestion.If
additionalpreserviceworkisrequiredofteachers,willitsbenefitslikelyoutweighthe
costsofindividualschoosingnottoteachbecauseofupfrontcosts?Ifclassesdifferentiate
studentsbyachievementlevelinscience,willthebenefitsofmoreappropriatecontentfor
agivenabilityoutweighthepotentialcostsofsignalinglowachievementofsomestudents?
Knowledgeofneedsandoptionsandsolidlogicsforincorporatingthisinformationto
predictpolicyeffectsareabaseforgooddecisionmaking,buttheyarenottheonly
importantfactors;twomorestandout.First,tomakegooddecisions,leadersneedthe
resources,especiallythetime,tocollectandprocesstheinformationavailabletothem.
Second,decisionmakersneedtosharethegoalsofthebroadercommunitywiththe
resultingincentivetoworktowardthesegoals.Muchrecentdiscussionhasfocusedonthe
needtoalignthegoalsofeducationdecisionmakers(teachersandprincipals)withthe
goalsofthecommunity.Forexample,accountabilitylawssuchasNoChildLeftBehind,as
wellaspay‐for‐performancereforms,aimtoalignthegoalsofteachersandschoolleaders
withthoseidentifiedinthepolicy,usuallyachievementgainsforstudentsinmathand
reading.
Whilethefivefeaturesofgooddecisionmakingareconstant(knowledgeofcontext,
knowledgeofoptions,anunderstandingoftheworld,timeandotherresourcesfor
processinginformation,andalignedgoals),theindividualmostsuitedtomakeagiven
decisionwillvarygiventhequestionandcon‐textinwhichtheyoperate.Thechoiceofwho
makesdecisionsisoneofthemostinfluentialeducationchoices.Improvementinthe
educationsystemmaydependpartiallyonincreasingtheavailableinformationand
refiningframe‐worksforprocessingthatinformation,butitalsodependsonwisely
choosingthepersonmakingthedecisions.Thereareanendlessnumberofdecisionsthat
mustbemadewithineducation.Wecanchangethesystembychangingwhomakeswhich
typesofdecisionsaswellthesupportstheyhaveformakingthosedecisions.
Educationdecisionsmaybemadebythecentralgovernment(thestateorfederal
governmentintheUnitedStates),localgovernment(districts),schools,teachers,families,
orarangeofotheractors.Considerthequestionofwhoteaches.Schoolscouldsimply
choosetheirteachersfromthefullpoolofindividualstheycanattract.Thedistrictcoulddo
thesame,orthecentralgovernmentcoulddothesame.Alternatively,thecentral
governmentcouldsetstandardsandschoolscouldchoosefromteachersmeetingthose
standards,ordistrictscouldfurtherrestrictthepoolavailabletoschools.Teacher
preparationprogramsorteacherorganizationsalsomaylimitthepoolthroughadmission
decisionsandcollectivelybargainedcontracts.
Theoptimaldistributionofdecision‐makingauthoritydependsonwhohasaccesstothe
fivefeaturesofgooddecisionmaking.Schoolsandfamiliesarelikelytohavethebest
informationabouttheirlocalneeds,thoughperhapslessinformationaboutthelong‐run
needsofstudentsastheyentertheworkforce.Someprincipals,superintendents,and
contentspecialistsinstatedepartmentsofeducationmayhavethebestknowledgeof
optionsavailableforthequestionsathand,thoughknowledgevariationacrossschooland
districtleadersislikelytobelarge,resultinginunequalqualityofdecisions.Similarly,
productiveframeworksforprocessingdatatoinformdecisionsmaybemorecommonin
oneoranotherleveloftheeducationsystem.Inawell‐functioningdemocracyorrepublic,
thegoalsofelectedofficialsmaybemorealignedwithsocietalintereststhanthegoalsof
thosefurtherfromthethreatoflosingelections,butthisisclearlynotalwaysthecase,both
becausepoliciescaninfluencegoalsatalllevelsandbecauseelectedofficialshave
alternativegoalsaswell.
Theadvantagesanddisadvantagesofdifferentlevelsofdecisionmakingarenotstagnant.
Advancesincommunicationandinknowledgeofwhatworkscouldimprovelocalaccessto
informationonoptionsandhigher‐levelaccesstoinformationonneeds.Accountability
systemscanrealignthegoalsoflocalactorssotheyaremoreinkeepingwiththegoalsof
thebroadercommunity.Whilethesechangesarefeasibleandoftenuseful,somechanges
areeasierthanothers.Itisdifficulttoimagineenoughcapacityatthefederallevelto
choosewhichteachersteachwhichclasses,orevenwhichteachersarebestforeachschool.
Itisalsounlikelythateachschoolwouldchooseitscapitalconstructiontomeetsafety
desiresofthecommunitywithouthigher‐levelstandards.Somedecisionsarebettermade
locallyandsomearenot.Whilepartofthegoalofeducationresearchistoinformdecision
makers,anequallyimportantgoalistoidentifywhichpotentialdecisionmakersarebest
suitedforeachtypeofeducationdecision.Shouldparentschoosetheschoolstheirchildren
attend,orshouldlocalorhigher‐levelgovernmentactorschoose?Shouldsuperintendents,
principals,orteacherschoosecurricularorinstructionalapproaches?
Ifwetakeasagiventhatthegoalofresearchineducationfinanceandpolicyistoinform
educationdecisionsforthebenefitsofstudents,eitherintheshortorlongterm,themerits
ofresearchshouldbejudgedbyitsabilitytodoso.Multipletypesofempiricalresearchdo
andshouldinformdecisions.
Therecentpushformorefactsandmorevalididentificationoftheeffectsofdifferent
approacheswastheresponsetoalackofbasicinformationtobuildlogicsthatsupport
effectivedecisionmaking.Inmanyareas,knowledgeabouteducationalneedsandoptions
canbegainedatrelativelylowcostandcouldmeaningfullyimprovedecisions.Avarietyof
researchapproachescanprovidethisknowledgeiftheyaredonewell.Apoorlyidentified
studyofthecausaleffectofaprogrammaytelluslittlethathelpsinmakingthenext
decision.Ontheotherhand,aperfectlyidentifiedcausalstudyofaprogramcancontribute
toadecision,butitisunlikelytoprovidefullinformationforanobviousdecision.Similarly,
descriptionsoftheworldcaninformdecisionsjustaseffectivelyascausalstudiesifthey
identifyneedsorleadtohypothesesthatchangethesetofconsiderationsinvolvedina
decision.Yet,likecausalstudies,descriptivestudiesalsocanbeunhelpfuliftheydonot
providetheinsightsneededtobetterunderstandneedsandoptions.Bothdescriptiveand
causalanalysescanbefacilitated.Ifprogramrolloutsweredesignedwithevaluationin
mind,learningabouteffectscouldbepartoftheimplementation,andtheknowledgebase
couldbebuiltwithouttheadditionalcostsoflarge‐scalestudies.Asmoreandmorevaried
informationonstudentsbecomesavailableforthepopulationofstudents,identifying
needsandprogresswillbeeasier.
Researchandscholarshipcansupportdecisionmakingnotonlybycollectingdatabutby
hypothesizingandsynthesizingknowledge.Forthevastmajorityofsizabledecisions,the
answerofwhatisbestforthegoalswehaveisnottotallyclear.Shouldwecreatemore
choiceintheschoolsystem?Shouldwehaveameritaidportionofourfinancialaid
system?Shouldpreschoolsfunctionwithinschoolsorshouldwemaintaintheeclectic
natureofearlychild‐hoodeducation?Shouldwestructurethecurriculummoreorless,
increaseemphasisonscienceormusic,spendmoretimeonphysicaleducation?Theseare
bigquestions,andevenifwesharegoalstheyarenoteasytoanswer.Whileformal
researchresultsprovideinformation,itisreallyhowthisformalinformationcombines
withotherinformationtohelpdecisionmakersthinkabouthowtheworldworksthat
affectshowtheywillultimatelymakedecisions.Therefore,changingthefactorsthat
decisionmakersconsidercanaffectdecisionsatleastaspowerfullyasadditional
informationaboutprogramsorstudents.Importantscholarshipasksustoconsiderfactors
notconsideredbefore,suchastheunintendedresponsesofteacherstoaccountabilityor
theunintendedresponsesofcommunitiestochangesinschoolfinancelaws.Once
highlighted,thesemechanismsbecomepartoftheconsiderationinfuturedecisions.
Finally,educationdecisionscanbeimprovedthroughresearchthatin‐creasesour
understandingofdecisionmakersthemselvesandtheiraccesstoresourcestheyneedfor
gooddecisionmaking—knowledgeofneeds,op‐tions,logics,resources,andgoals.
Improvementintheeducationsystemmaycomemorequicklyandmoreeasilybychanging
whomakesdecisionsthanbychangingthegoalsandskillsofthosecurrentlydeciding.
Theuseofdataindecisionmakingisrarelytransparent.Ignoringwhatseemtobefactsin
thechoiceseducationleadersmakemayinfactbetheresultofconceptions—developed
withthesupportofdataandresearch—abouthowtheworldworks.Theroleof
researchers,then,isbroaderthanprovidingpiecesofinformation.Itistosupporteffective
decisionmakingbyprovidinginformationonneedsandoptionsaswellasframeworksfor
understandingtheworld.Itisalsotoidentifyeducationsystemsthatgivedecision‐making
authoritytoindividualsmostlikelytohaveaccesstotheresourcesneededtomakegood
decisions.
REFERENCES
InstituteofEducationSciences.n.d.AboutIES:Connectingresearch,policyandpractice.
Availablehttp://ies.ed.gov/aboutus/.Accessed17October2011.
CenterforDataDrivenReforminEducation.2011.www.cddre.org.Accessed17October
2011.
SpencerFoundation.2010.Datauseandeducationalimprovement.Available
www.spencer.org/content.cfm/strategic‐initiatives.Accessed17October2011.
U.S.DepartmentofEducation.WhatWorksClearinghouse.n.d.http://ies.ed.gov/
ncee/wwc/.Accessed17October2011.