MedbiqxAPIWorkshopReport
Authors:
DavidTopps1,CoreyAlbersworth2,EllenMeiselman3,MaureenTopps1,SarahTopps4,
SandraMorrison1
1UniversityofCalgary
2AthabascaUniversity
3UniversityofMichigan
4SimonFraserUniversity
Background:
Howdoweknowourlearnersarecompetent?Uponwhatdowebasethis?Observations
andjudgmentsfromteachers,usingITERsandMCQsformthebasisofthevastmajorityof
ourassessments.Whiletheseworkwellforthemajorityoflearnersandteachers,weare
lesseffectiveatearlydetectionofmaladaptivelearnersorthosewhoarestruggling.We
tendtogivethebenefitofthedoubt,thinkingthisakindness,butdelayeddetectionand
diagnosisoflearnersindifficultyisdoingnobodyafavor.
Inourhighschoolmathtests,weareconcernedthatlittleJohnnynotonlygottheright
answerbutusedtheappropriatestepsandtechniquesinderivingthatanswer.Similarly,in
medicaleducation,weshouldbeconcernedwithassessmentofmorethanmemorization,
andlookatproblemsolvingandefficient,effectivetaskcompletioninavarietyofcontexts.
Weshouldlookatwhatourlearnersdo,notwhattheyortheirteacherssaytheydo.Hence,
theriseininterestinactivitymetrics.ThisconceptisatthepeakofMiller’sPyramidof
Assessment(Miller,1990)demonstratingprofessionalauthenticity,movingfrom“knows”
to“knowshow”to“showshow”to“does”(althoughdoubthasbeenraisedaboutthe
evidencetosupportMiller’spyramid,butwedigress(Lalley&Miller,2007)).
Thisisnotanewconcept,butourabilitiestomeasuresuchactivitieswiththenecessary
richnessofdetailhaslargelybeencompromisedbycostandpracticalityandhence,werely
oncompoundobservationsandjudgmentsofhumansensors–ourteachers.Butnow,with
theplethoraofcheapdevicesandsensors,thepossibilitiesforgatheringmeaningfuldata
aboutthelearningprocesshavegreatlychanged.
Inourpreviousstudies,usingmixedsimulationmodalitiesinourHealthServicesVirtual
Organization(HSVO)project,wewereabletocombinemultiplelearningtoolsandsystems,
usingasophisticatednetwork-enabledplatform.Wecombinedvirtualpatients,highfidelitysimulators,low-latencyvideocollaboration,virtualanatomymodels,andlight-field
cameraarraystogeneratevirtualpointsofview.Wewereabletocreatepowerfullearning
designsforcollaboratinggroupsoflearnersacrossmultipledisciplines,institutions,and
continents,inreal-timewithmanyconcurrentdatastreamstrackingtheireveryaction.
Bothlearnersandteachersenthusiasticallyadoptedthesemulti-modalitysimulation
designs.Weobservedaprogressionofskillsdevelopmentandcollaborativeproblem
solvingapproachesinallthegroups.Analysisoftheactivitystreams,whichwaslargely
basedonmanualcodingofvideotapedactivities,provedtobeextremelylaborious,witha
highcostineffort.Wedidobserveintenseclustersofactivity,centeredonboundary
changesincontext,somewhatanalogoustophasechangesinsimplerbiologicaland
physicalprocesses.Thevigorousup-spikeinsuchactivitieswasoftenhardtokeepupwith,
andsometimesmissedbyboredassessorsskippingthroughlongperiodsofminimalstate
change.Humansdonotdowellwiththisapproach–machinesensorsaremuchmore
reliableandefficientinsuchmonitoring.
TheHSVOprojectdemonstratedsomeextraordinarylearningactivitiesbutwasavery
costlyendeavor,withhighinfrastructureneeds(user-directedlightpipes,high
performanceclustercomputing),highlyskilledpersonnel,anddependenceonhighlevels
oftechnologywithlimitedportability.Whilethecoremiddlewarelayer,Savoir,was
designedtoactasanintermediarybetweenmanydifferentdevicetypesandsystems,it
wasnotveryflexibleandrequiredanoverlytightdegreeofintegrationandbinding
betweenApplicationProgrammingInterfaces(APIs).
Inmanyways,thiswassimilartotheproblemsencounteredwithattemptingtoblend
informationstreamsacrossothereducationalplatformsusingSCORM.Itwasabold
attemptatthetimetointegratedatafrommanysourceswiththecoreLearning
ManagementSystems(LMSs).ItswasdependentonthecentralstructureoftheLMS-an
architecturalmodelwhichisnotparticularlysuitabletocurricularstructureofmost
medicalschools.ItsrestrictivedatastructuresmadeSCORMvulnerabletoalltheongoing
changesateachsideofeverysysteminterface.
Thisverychallenge,facingtheadoptersofSCORM,drovethedevelopmentofthe
ExperienceAPI(xAPI,alsoknownastheTinCanAPI).CommissionedbyAdvanced
DistributedLearning(ADL),itwasoriginallydevelopedbyAndrewDownesandteamat
RusticiSoftware.xAPIhasamuchsimpler,moreflexibleandopenstructurebasedon
Subject-Verb-ObjecttripletsthataresimilartotheResourceDescriptionFramework
(RDF).
Topromotetheadvanceofdatainteroperabilitystandardsinthisareaofactivitymetrics,
theMedbiquitousLearningExperienceWorkingGroupwasformed.ThisgroupchosexAPI
astheircentralmechanismtocoordinateandconnectsuchactivitymetrics,butisalso
monitoringothersimilarprotocols.Thegrouphasbeenworkingwithkeyplayersand
organizationsrelevanttoxAPI,withacademic,governmentandindustryrepresentation.
Thisreportdescribestheactivitieswithinandleadinguptoanintenseworkshopabout
xAPIandblendedlearningattheMedbiqannualconferenceinBaltimore,inMay2016.This
reportassumessomefamiliaritywiththeprinciplesandprocessessurroundingtheuseof
theExperienceAPI.Therearemanyotherexcellentarticlesoutthereexplainingthesebasic
principlesbetterthanwecaninthespaceavailablehere.
ThisreportalsodoesnotfollowthetraditionalIMRADlayoutofasingleresearch
intervention.Itisintendedtobedescriptiveofthedesign-basedresearchapproachthatwe
tookinourdevelopmentofthetoolsandresourcesthatculminatedinourworkshop.See
Discussionformoreonthis.
DevelopingxAPIProfiles
TheExperienceAPIissimpleinconcept:theSubject-Verb-Objectconstructhasbeen
describedas:
Bob Did This
But,inorderforthexAPIstatementstobemeaningfullycomparedacrosssources,when
theyarestoredintheLearningRecordsStore(LRS),thereneedstobesome
standardizationaroundthevocabulariesused.Withinthissimplestructure,
standardizationoftheVerbsused,alongwiththeirmeaningandcontextisthefirstthingto
tackle.
Groupsofverbs,andotherusagerequirements,thatareassociatedwithacommonsetof
activitiesarecombinedintosetsknownasProfiles.TheMedbiqLearningExperience
WorkingGrouphastakenonthetaskofdefiningaseriesofProfilestosupportmedical
educationactivitiessuchassimulationsandscenarios.Thegroupchosetostartwiththe
VirtualPatientxAPIProfile(Topps,Meiselman,&Strothers,2016),astherealreadyexists
someexcellentbaseworkaroundtheMedbiqVirtualPatientdatastandards,andthe
relatedactivitiesarerelativelywellconstrainedwithgoodcommonalityacrossplatforms.
ThegroupplanstocontinuetodefineaseriesofProfiles,relatingtomannequinsandtask
trainers,standardizedpatients,scenariosandblendedsimulations,andvirtualworlds,in
collaborationwithotherMedbiqWorkingGroupssuchasCompetencies,alongwithother
partnersandinterestedorganizations.
Developingastandardanddesigningtheprofilesthatdescribeitrequiresacorestructure
andtheoreticaldesign,butifthisisnotgroundedinthepracticalitiesofimplementation,it
iseasyforthestandardstobecomeisolatedandesoteric.Accordingly,Medbiqhasbeen
veryactiveatpromotinglearningsessionsandworkshopsaroundxAPIoverthepast3
years.Forthisyear’sworkshop(Meiselman,Topps,&Albersworth,2016),weassembleda
learningdesignthatwouldintegrateabroadvarietyoflearningactivities,withactivity
metricsderivedfrommultipleconcurrentsources.
Thisendpointandtimelinegreatlyfacilitatedamoreefficientandcollaborativeapproach.
ItalsogeneratedmuchinterestinthexAPIcommunityandwewerefortunatebeneficiaries
ofmanycollaborativeactivitiesandproblemsolving,farexceedingourexpectations.
Frankly,wewereastoundedbyhowhelpfuleverybodywasinmovingthingsforwards.
ThisisdefinitelyagreatstrengthofworkingwithxAPIatpresent.
Developingaworkshop
BasedonourexperienceswiththeHSVOProject,anditspowerincombiningmultiple
simulationmodalitiesintoeffectivelearningscenarios,wecreatedasimplescenariothat
woulddemonstrateawidevarietyofactivitymetricsfrommultipleconcurrentsources.
TocontrastwiththecomplexityandcostoftheHSVOProject,wealsochosetobaseour
learningdesignonmuchsimplerandcheaperdevices.HSVOcostmorethan$2million;the
average“disposableincome”foraneducationalresearchprojectismorelike$5-10k,sowe
wantedtodemonstratewhatispossibleonamuchmorerestrictedbudget.
Inparticular,wewereinterestedinexploringwhatcouldbedonewiththecheapsensors
andsimplecomputingcoresavailableontheArduinoplatform,whichisdesignedfor
hobbyistuseandverylowbudgets.ThereareothersimilardevicessuchasRaspberryPi,
whichalsohavegreatpromise,butaquickenvironmentalscanshowedArduinotobethe
mostsuitabletoourneeds.
InourHSVOProject,wefoundtheopen-sourcevirtualpatientplatform,OpenLabyrinth,to
beaveryeffectiveintegratorofsimulationrelatedactivities.Itwasexcellentatproviding
flexible,easilymodifiable“contextualglue”toholdthevariouscomponentsofalearning
scenariotogether.Tightcoupling,usingtheSavoirmiddleware,provedtooinflexibleinthe
HSVOProject.ThemoreopenapproachaffordedbyxAPIwasmorepromising.
DevelopingaQuiverofxAPIsources
ThisentailedtheincorporationofxAPIintoOpenLabyrinth.Wewerefortunateintwo
ways:wehadsecuredfundingviaaCatalystGrantfromtheO’BrienInstituteofPublic
HealthattheUniversityofCalgarytoextendOpenLabyrinthwithimprovedactivitymetrics
andenhancedintegrationwithotherresearchplatforms;andtheinherentarchitectureof
OpenLabyrinth,whichalreadycontainedausefulsetofinternalactivitymetrics,andwas
highlyconducivetointegrationofxAPIstatementoutput.Wewerepleasedtoseethatthis
waspossiblewithlessdevelopmenttimethanwehadbudgetedfor,andinashorter
timelinethananticipatedbecausetheexistingdatastructuresweresogood.
WebasedthexAPIstatementformattingontheMedbiqxAPIVirtualPatientProfile,and
alsotooktheopportunitytoreflexivelyimprovetheMedbiqprofilebyconsideringsomeof
thepracticalimplicationsastheypertainedtoourcasesandscenarios.Thistwo-way
reciprocaldevelopmentwascarriedoutwithfullcollaborationwithotherworkinggroup
membersandthexAPIcommunity.AllsourcecodeisfullydocumentedonGithubat
https://github.com/olab/Open-Labyrinthandwearehappytocommunicatewithother
developmentgroupsaboutsomeofthesmallchallengesthatwefacedinour
implementation.
Whilewewereexploringotherwaysinwhichwecouldcaptureactivitymetricsinour
scenario,usinganumberofdifferentcollectionmechanisms,wecameacrossthe
GrassBladexAPICompanion(www.nextsoftwaresolutions.com/grassblade-xapi-companion),a
simpleusefulutilitythatprovidesxAPIstatementsfromWordPressactivities.Sincewe
alreadyuseWordPressforourOpenLabyrinthsupportsiteandblog,thispresentedan
idealopportunitytoextendouruseofwebplatformsforadditionaldatagatheringand
activitymonitoring.
WethenimplementedtheGrassBladeLRSonourservers.Wewerepleasedtonotethatthe
GrassBladeLRSalsousesMySQLforitsinternaldatabase,asdobothOpenLabyrinthand
WordPress.Thisgreatlysimplifiedsomeoftheintegrationandtestingofdatatransfer
betweenthesesoftwareapplications.Wehadbeenquiteconcernedaboutwhattypeof
databaseinfrastructuretouse,asourinitialreadingsonthetopicofLRSsseemedto
suggestthatanoSQLdatabase,suchasMongoDB,wasdesirable(Abbey,2016)(Kaplan,
2014)(Korneliusz,2014;Mei,2013).Sinceourdevelopmentteamsweremuchmore
familiarwithSQL,thispresentedasignificantpotentialbarrier.
TherewasathirdfactorwhichwefoundattractiveforchoosingGrassBladeasourprimary
LRSfortesting.ThereisaflatpricingstructureperLRSinstance,withoutadditionalcosts
basedonthenumberofstatementsstored.Inourearlyphases,wehadverylittleideaof
thenumberofstatementsthatwewouldbestoringorofthesizeofthedatabaseswe
wouldgenerate.WewerealarmedtohearofanothersimilarprojectatapreviousMedbiq
conferencethatmanagedtogenerate300,000xAPIstatementswithin36hours,which
createdasizeablebill–somethingourfunderswerekeentoavoid.
Forourinitialtesting,anotheradvantageofthisapproachbecameapparent:itwassimple
togeneratexAPIstatementsfromourWordPresssite(http://openlabyrinth.ca)and
examinetheminourGrassBladeLRS.Thisgaveusgreaterpracticalfamiliaritywiththe
formattingofxAPIstatementsandsomeofthesyntacticalrulesimposedbytheLRS.We
werefortunateinthattheGrassBladeLRSisrelativelyforgivinginitsxAPIstatement
interpretation,whichaffordedgreaterexperimentationwithdatasourcesintheearly
phases.
DuringourexplorationsofWordPressandxAPIstatementgeneration,wealsocameacross
H5Pwidgets(https://h5p.org).TheH5Pprojecthasbeendevelopinginteractivewidgetsthat
canbeincorporatedintoanumberofplatforms,suchasWordPress,Moodle,andDrupal.
TheinitialattractionisthatmanyoftheseH5PwidgetsgeneratexAPIstatementsoftheir
own.Forthis,theyrequireasimpleintermediaryframework–thereisoneavailableasa
WordPressplugin.ThismeantthatwewereabletoimplementanothersourceofxAPI
statementswithlessthananhour’swork.
Wethennotedthat,inadditiontobeingopen-source,H5Pmakesiteasytoincorporate
theirintermediarysupportingframeworkintoone’sownapplication.Thiswasidealfor
incorporatingH5PwidgetsintoOpenLabyrinthandthiswasaccomplishedwithonlya
modicumofdevelopertime,givingusyetanothersourceofxAPIstatements.
TheH5Pwidgetdesigninterfaceisverysimpletoworkwith,allowingustorapidly
generateanumberofinteractiveuserinterfaceextensions,extendingtheutilityofboththe
WordPressandOpenLabyrinthplatforms.Wewerealsodelightedtodiscoverthatwidgets
createdforoneplatformareeasilyportabletoanother,furthermultiplyingour
developmentefforts.
DevelopingSensorHardware
FromthesemultipleexperimentswithgeneratingxAPIstatementsfromallthesesources,
wewerethenmoreeasilyabletovisualizehowonemightgeneraterawxAPIstatements,
usingsomebiometricsensorsandtheverysimpleArduinohardwareplatform.
Initially,thiswasonlyintendedtobeaproofofconceptdemonstrationofthefeasibilityof
generatingsuchxAPIstatementsandthesimplicityoftheirformatting.TheArduino
platformisrenownedforbeingcheapandeasytoworkwith,andalreadyhasawide
varietyofcheapsensorsthatarelargelyplugandplay.
Inaccordancewiththeoveralllearningdesignfortheworkshop,wherewehopedto
demonstratetheabilitytomeasureabroadvarietyofconcurrentactivities,weselected
sensorsthatmightgiveussomecrudeindicationofstresslevelsintheparticipants.We
anticipatedthatthrowingsuchaplethoraofactivitiesintothemixmightbecomequite
chaotic(makesforafunworkshop!)andhardtofollow.
Figure1:showingtheArduinosensorsinuse(lefthand)whilethesubjectplayedanOpenLabyrinthvirtual
patient.Rightscreenshowssensoroutputinreal-time
Forthebiometricsensors,wechoseacombinationofheartrateandgalvanicskinresponse
(GSR).Thesearewellknowntobeassociatedwithstresslevelsandare,indeed,twoofthe
parametersusedinaliedetector.Whatsurprisedusinourinitialtestingwasthe
sensitivityoftheseindicators.
Withaconveniencegroupofco-investigatorsintheproject,weexaminedhowtheheart
rateandGSRvariedastheynavigatedachallengingOpenLabyrinthvirtualpatientcase.For
thecases,wemodifiedtwoexistingVPcases,Gail’sDilemma
(http://demo.openlabyrinth.ca/renderLabyrinth/index/727) andRushingRoulette
(http://demo.openlabyrinth.ca/renderLabyrinth/index/723).Therewerealreadysometime
pressuresandintensityinherentinthesecasesbutwedecidedtouptheantebyusinga
newfunctioninOpenLabyrinth,withtimer-enforcedpopupwarningmessagesand
navigationaljumpspropellingthemonwardsatanever-increasingpace.Thefirstcase
generallylastsabout15minutes,whereasthesecondcasesetof20mini-casescanusually
becompletedwithin5-8minutes.Thisgaveussufficienttimetoobservechangesbutwas
shortenoughtobeabletorunmultipleiterationswithoutparticipantburnout.
Ourgroupconsistedofthreeexperiencedphysicianteachers,apublichealthprofessional
withextensivevirtualpatientexperience,andacomputersciencestudentfamiliarwiththe
platformperformance.Wevideotapedthesessionforlateranalysis,whilewerecorded
theirperformanceontheVPcases.Wealsokeptindependentfieldnotesduringthesession
forlatertriangulationofobservations.Weconnectedeachparticipanttotheheartrateand
GSRsensorsthenwatchedthechangeswhiletheyeachnavigatedtheVPcasesinturn.
Itwasfascinatingtonotehowthedifferentparticipantsrespondedtothestressofthe
cases.Asexpected,withtheirvaryingbackgrounds,eachfounddifferentelementsoftheVP
casestobechallengingindifferentways.Allparticipantsfoundthattheincreasinglytight
timingwasstressfulorannoying.Normally,thetimeintervalallowedfordecisionsonthe
mini-casesis30seconds.Forthisseries,westartedat25seconds,subtractedasecondfor
eachsubsequentcase,leadingtoonly6secondsforthefinalcase,whichisbarelyenough
timetoreadit,letalonemakeaninformeddecision.
Otherfactorsinthecaseplaywereapparentlystressfultoeachparticipantindifferent
ways.Somedislikedhavingtomakedecisionsbasedonincompletedata;somefoundthat
interfacequirksthatarosewerequiteannoying.Butwewereeasilyabletodetectthese
changesinstresslevel,evenwhenquitesubtle.Oneparticipant,wellknownforcool
performanceunderpressure,maintainedaremarkablyconsistentsetofparameters,until
thetime-outcrossedthepointofreasonablereadabilityforthecase.Assoonasaforced
jumpkickedin,therewaschangeinbothHRandGSR.
Twooftheparticipantsinthisgroupwereonsignificantdosesofbetablockersforan
unrelatedmedicalcondition.Weanticipatedthatthismightbluntthestressresponseand
limittheusefulnessofthesensors.Whilebothofthemhadverylowrestingheartrates(58
and62beatsperminute(BPM)respectively),thesensorswerestillabletoeasilydetecta
changeintheirstresslevelsastheychallengedthecases.
Weweredelightedbytheseearlyfindings,whichweremuchmoresensitivethanwe
suspected.Despitethemildbutintentionalstressorsinducedinthisscenario,all
participantsreportedthatthecaseswereengagingandtheexperiencewasenjoyable.They
wereintriguedtoseecleardemonstrationthattherewasanapparentlytightassociation
betweentheirmildstresslevelsandsensorfindings.
DevelopingthexAPISensorInterface
Allthestepsuptothispointhadbeenfairlyeasytoimplementandwerestartingtoshow
aninterestingconfluenceoffactorsanddata.Weanticipatedthatcompletingtheinterface
betweensensortrackingandtheLRSwouldalsobequitesimple.
Weenlistedtheassistanceofcomputersciencestudent(CA)incodingtheArduinosensor
interfaceandxAPIintegration.Thiswastobepartofaself-learningproject,with
recognitionforcoursecredits.
UsingtheProcessingIntegratedDevelopmentEnvironment(PIDE),theinitialstepswith
thesensorandtheArduinoenginewerequitestraightforward.UsingtheArduinotosend
outvaluesviatheserialporttotheprocessingPIDEwithacharacterinfrontoftheValue
sothatPIDEknewwhatsensorthevaluewascomingfrom.PIDEwouldthendisplaythis
withavisualizationthatwouldshowtheheartwaveformandBPMvalues.Duetothe
limitationsofPIDE,itwasunabletosendaproperlyformattedHTTPpostrequesttothe
LRSwithaJSONstatement.WehadtoconvertthePIDEcodetothemorepowerfulEclipse
IDE.VariousadditionaltoolswereneededtogetthePIDEcodeupandrunningonthe
EclipseIDE,includingimportingalloftheproperlibrariesfromPIDE.
Figure2:PIDEoutputwindowshowingheartrateandGSRoutputdatafromtheArduinosensors
NowthattheprogramwasfunctioninginEclipse,wewereabletousetheexistingxAPI
librariestosetupaclientthatwouldsendxAPIstatementstotheLRSatanyintervalthat
wechose(inthiscaseeveryfiveseconds).Thesevalueswouldthendisplaysidebyside
withwhatthelearnerwasdoinginthetestcase.
{
verb : {
id : http://adlnet.gov/expapi/verbs/imported,
display : {
en-US : imported
}
},
actor : {
name : medbiq,
mbox : mailto:info@openlabyrinth.ca,
objectType : Agent
},
object : {
id : http://demo.openlabyrinth.ca,
definition : {
interactionType : choice,
choices : [
{
id : http://demo.openlabyrinth.ca,
description : {
GSR:3219 : BPM:153
}
}
]
}
},
id : a7eb9501-d0ae-4cf9-adbb-a912439f7727,
stored : 2016-05-17T18:29:36.140Z,
timestamp : 2016-05-17T18:29:36.140Z,
authority : {
account : {
homePage : http://openlabyrinth.ca/grassblade-lrs/xAPI/,
name : 14-1e2c639ccc936c7
},
objectType : Agent
}
}
Figure3:examplexAPIstatementgeneratedbyourArduinodevice
PleasenotethatFigure3isforillustrativepurposes.Itislikelytherewillbefurther
changestotheverbsusedbythedevice,andthattheinteractionTypewillbechangedfrom
‘choice’to‘device’.Atpresent,theProfilesinthisareaarestillevolving.
WehadafewsmallproblemswithxAPIstatementformattingbutwithhelpfrommembers
ofthexAPIcommunity,wewereabletosuccessfullylinkourGrassBladeLRS.Inparticular,
weowemanythanksandwishtoacknowledgetheextensivehelpwereceivedfromPankaj
AgrawalatGrassBlade,AndrewDownesatRustici,andCoreyWirunatCardinalCreek.
TheformattingofourxAPIstatementsfromadevicestillneedstoberefined.Butthe
importantaspectherewasthattheoutputwasstillusefulandusableinitscurrent
temporaryformat.
OtherpotentialxAPIsources
Anumberofothersourcesofactivitytrackingwereconsidered,exploredandpartially
incorporatedintotheworkshopscenariodesign.Forexample,weinitiallyexploredthe
potentialuseoftheTobiirangeofeye-trackingsensors.Tobiimakesaverysophisticated
rangeofsensorsforresearchpurposesbutmostofthesearequiteexpensive.Oneofthe
learningdesignparametersthatwechoseforthisworkshopwasaccessibilityand
affordability.WehadhopedtousethesimpleroutputsfromtheTobiiEyeXController
(www.tobii.com/xperience),whichismuchcheaper.However,thecompanywas
overwhelmedwithdemandforthislevelofsensorandwasnotabletoassistuswiththis
integration.Thismaybeworthexploringinfuture.
Wealsonotedthatitisnowquitepossibletoblendtheactionsandeffectsofanumberof
web-basedapplications,usingsoftwarelikeZapier(https://zapier.com)andIfThisThen
That(IFTTT--https://ifttt.com).Wewereabletocreatesomeinterestingintegrations
betweenWordPress,Instagram,Evernote,Twitterandthesmartphonecameraitself,in
waysthatcanfurthergeneratexAPIstatementstobestoredinourGrassBladeLRS.While
integrationsareverysimpletoimplement,inthefinalsetupoftheworkshop,wedidnot
relymuchontheseintegrations.Theydidprovidesomeoftheworkshopparticipantswith
additionalmeansofcontributingandactivelytracking,withtheuseofxAPIstatements,the
complexactivityflowsofthisverydynamicsession.
WewerecaughtoutbyadiscrepancyinthebusinessmodelofZapier.Fordemonstration
purposes,weintendedtosetupthefivefreeZapierinteractionsthatareadvertised.
However,wefoundthatZapier’sbillingstructuresrequiredpaymentassoonaswesetup
thesecondinteractionchannel.Wedidnothavetimetoexplorethisbillinginconsistency
anddidnotconsiderthatthevalueaddedwasworthitatthispoint,sowesimply
continuedwiththefreeIFTTTservice.
Figure4:simpledataflowdiagramshowinghowxAPIstatementswereusedbetweendifferentapplications
DeployingtheWorkshop
AllofthesemultipleactivitiesanddatasourcescametogetherJustInTime.Weare
particularlyindebtedtoCAforanextraordinaryeffortinbringingtogetherthepieces
neededformeaningfulxAPIstatementsfromourArduino-basedsensors.Hewasableto
coordinateandcollatemultiplestreamsofadvicefrommanyinthexAPIcommunity.
Sometimesthereisnotalotofsleeponthebleedingedge.
FortheMedbiqworkshop,wewereabletodemonstratealiveworkingexampleoftracking
activitydatafromthefollowingresources:
1. Heartrate,viaArduinosensor
2. GSR,fromArduinosensor
3. MouseclicktimingsfromOpenLabyrinth
4. DecisiontreeresponsesfromOpenLabyrinth
5. QuestionresponsesfromOpenLabyrinth
6. ForcednavigationjumpsandtimeoutsfromOpenLabyrinth
7. H5PwidgetinputfromOpenLabyrinth
8. H5PwidgetinputfromWordPress
9. EvernotenotesviaWordPress
10. IFTTTDOCameraimagesviaWordPress
Inthespiritofengagingasmanyworkshopparticipantsaspossible,aswellastracking
activityandstresssensorsonourvolunteers,wewerealsoabletoengageactive
contributionsandlearningactivitiesfromparallelparticipantsastheyusedtheadditional
datasourcestocontributetotheoveralldatastream.
Theworkshopwassuccessfulinaddressingawidevarietyofparticipantneeds.Wehada
mixofthosewhowereprimarilyinterestedinthetechnicalaspectsofxAPI;thosewho
wereinterestedinexploringwhatcouldbedonewithsuchmetrics;andthosewhowere
interestedinexploringtheenhancedvarietyoflearningdesignsthatcouldbesupportedby
suchamulti-modalityapproach.
Thereappearedtobeaveryhighdegreeofengagementinalllevelsofthevarious
activities,whichsometimesmadeitdifficulttokeepeveryoneontrack.Whentimewasup,
theroomwasslowtoclearbecauseofthecontinuinglevelofveryactivediscussion.Note
thatthiswasoneofthefinalsessionsoftheconference,whenmostpeoplearestartingto
flag.
AnalyzingtheLRSData
AssoonaswestartedcollectingxAPIstatementsinourGrassBladeLRS,wewereableto
performsomebasicanalysisusingitsbuilt-intools.
Figure5:exampledashboardfromGrassBladeLRS
Aswenotedearlier,GrassBladeisveryforgivinginitsparsingofxAPIstatements.This
makesthingsmucheasiertosetup,ratherthanjustsloggingthroughalongseriesoferror
statements,whichweverymuchappreciated.Wealsogreatlyappreciatedtheaccessibility
ofGrassBlade’screator,PankajAgrawal,whowasveryhelpfulintroubleshooting
throughoutallphasesoftheproject.
WewerealsoveryfortunatetoreceivemuchhelpandadvicefromRusticiSoftware.They
provideduswithfreeaccesstosandboxaccountsonboththeirSCORMCloud
(http://scorm.com/scorm-solved/scorm-cloud-features)andWatershedLRSs
(www.watershedlrs.com).SCORMCloudhasanumberoftestingfunctionswhichmakeit
easierfordeveloperstofinetunethesyntaxoftheirxAPIstatements,includingamanual
xAPIGenerator(https://tincanapi.com/statement-generator).
Figure6:exampledashboardfromWatershedLRS
TheWatershedLRSisamuchmorepowerfulanimal,withserioushorsepowerand
powerfulvisualizations.Thiswaslargelyoverkillforthesimpleaspectsofthisprojectand
wellbeyondthebudgetavailable.However,AndrewDownesfromRustici,andPankaj,
wereextremelyhelpfulinfinetuningtheinterfacesintheirrespectiveLRSssothatwe
couldexploredirectdataintegrationbetweenLearningRecordStores.Thisgaveusaccess
tosomedatavisualizationsavailableinWatershed,bydirectlypullingxAPIstatements
fromtheGrassBladeLRS,andallowedustoexplorethepowerofbigleagueLRS,whilestill
usingthesimpleflexibilityofGrassBladeforourinitialdatacapture.Wecannotapplaud
enoughthecollaborationevidentinthexAPIcommunityinsurmountingthechallenges
thatwefoundintakingthismulti-layeredapproach.
Webrieflyexploredthehigherlevelsofassessmentbeyondactivitymetrics,basedonxAPI
statements,thatmightbeaffordedbyMozillaOpenBadges(http://openbadges.org).The
WatershedLRSdoesprovidesomebasicfunctionalitythatsupportsintegrationofBadges.
However,onfurtherdiscussionwiththeRusticiteam,itwasclearthatfewinmedical
educationareinterestedinusingMozillaBadgesatthispointandthereforeweredirected
ourefforts.
WewerealsofortunatetohaveaccesstoafreetrialaccountontheSaltboxWaxLRS
(www.saltbox.com).Weinitiallythoughtthatitwouldjustbeinterestingtosimultaneously
sendxAPIstatementsto3differentLRSsfromOpenLabyrinth.Thiswassurprisinglyeasy
tosetupanddidnotexacttoomuchofaperformancehitonourOpenLabyrinthserver.It
hassincebeenpointedoutthatthisisasomewhatredundantapproach.Itisinthenature
ofLRSstobedirectlyfederated.SoitwouldbemoreeffectivetouseLRStriggersandfilters
tosendselectedstatementsfromoneLRSontoothers,ratherthantoexpectthesource
softwaretohandlestatementsendingtoall3LRSs.
ThetestingwiththeWaxLRShadanadditionalbenefit:itismuchmorestrictinitsparsing
ofxAPIstatements.Aftersomeinitialgrumblingonourpart,wequicklyrealizedthe
benefitsofthis:itenabledourdeveloperteamstobemuchmoreexactintheirxAPI
statementparsing,whichresultedinamuchcleanersetofxAPIstatementsfrom
OpenLabyrinthandourH5Pwidgets.WethanktheSaltboxteamfortheirassistanceinthis.
OpenLabyrinthitselfhasquitesophisticatedinternaltrackingandreportingofactivity
metrics,alongwithsomesimplevisualizations.
Figure7:pathwayanalysisreportgeneratedinternallybyOpenLabyrinth
Indeed,theOpenLabyrinthdeveloperteamwasinitiallyperplexedastowhywewere
introducingtheadditionalcomplexityofxAPIstatementreportingsince,atthatstage,we
wereabletoachievethesamelevelofactivityreportingusingitsownfunctions.However,
sinceweimplementedxAPIreportingdirectlyintoOpenLabyrinth,severaladvantages
havebecomeclear.
Firstly,ithasaffordedsimultaneousactivitytrackingacrossamuchwiderrangeoflearning
activities,notjustOpenLabyrinth,asdescribedintheparagraphsabove.Secondly,ithas
relievedourdeveloperteamofthecontinuingburdenofcreatingcustomreportsfor
variousresearchgroups.Thisisthebaneofmanydatabasedresearchtoolsandcan
becomeahugedrainonresources.Butthirdly,andmostimportantly,ithasgivenusaccess
toamuchwiderrangeofdatavisualizationtools.
WiththeintegrationofxAPI,wehavenowmovedintotheBigDataworldinlearning
analytics(Topps,Meiselman,Ellaway,&Downes,2016).Westatethis,basednotuponthe
Volumesofdatawearegenerating(althoughthesearerapidlyincreasing),butmoreupon
theotherVsthatarecommonintheprinciplesofBigDataanalytics:Velocity,Variety,
VeracityandVisualization(Kobielus,2013).
Byplacingouractivitymetricsinacommonformat,withinanLRSthatisdesignedto
acceptlargevolumesofdatafrommultiplesourcesandfederatethemwithotherservices,
wecannowtakeadvantageofsomeofthemorepowerfulvisualizationandanalytictools
beyondtheworldofmedicaleducation.
ItalsoallowsustoexplorebothpowerfulcommercialvisualizationtoolslikeTableau
(www.tableau.com),whichisdesignedforusewithverylargedatasets,andalsocheapbut
versatilevisualizationlibrarieslikeD3.js(https://d3js.org),anopen-source,componentized
visualizationlibrary.
Discussion
ForthisTechnicalReport,wehavenotadheredtotherigidIMRADstructurecommonto
mostacademicarticles,inthebeliefthathighlightingsomeofthedesignissuesthatarose
duringthedesign-basedresearchelementsoftheprojectwasmoreconsistentwithour
discoveryorientedapproach.Hence,severalofthediscussionpointshavealreadybeen
coveredearlier.Wewouldhoweverliketofurtherhighlightsomeimportantpointsthat
arosefromtheprojectasawhole.
ItwasveryclearthattheloweredcostofintegratingdifferentsystemsusingasingleAPI
thattrackseventsandmetadataisabigadvantageofusingxAPI.WefoundthatxAPImakes
itpossibletoadaptlowcost,opensourcetoolsforsensorsandlearninginteractions,and
wecannowtrackdatathatmayilluminatethelearningprocessfarbeyondwhatwas
possiblewithSCORM.Whatwedidinmeasuringstressparametersinlearners,while
engagedinaneducationalexercisewasnotparticularlynew(Hardy,Mullen,&Jones,
1996)(Aroraetal.,2010).Butpreviouseffortshaverequiredasophisticatedsimulation
laboratorywithexpensivesuitesofsensorequipment,whicharelargelybeyondthebudget
ofmosteducationresearchers.Otherattemptshaveusedsubjectivemeasuresofperceived
stress(Cohen,Kamarck,&Mermelstein,1983)which,whilestillrelevant,havetheirown
biasesandlackofsensitivity.Ourapproach,usingsimplesensorsandcheapcomputing
devices,alongwithaflexibleapproachforgatheringactivitymetricsfrommultiplesources
representsaninterestingadvanceinthisarea.
Conclusions
Datafrommultiplesourcesgreatlyextendsourabilitytotrackwhatlearnersactuallydo,
notwhattheysaytheydoorwhattheirteacherssaytheydo,thuspotentiallyreducing
manysourcesofbias.
LooseaggregationofactivitiesviaxAPIismorepracticalthanthetightcouplingpreviously
envisionedbySCORMasatranslationmechanism.Bylooselycouplingthedataflows,we
expectthattherewillbegreaterflexibilityinlearningdesignsandlessdependencyon
keepingstructuralchangeswithincommunicatingsystemssuchasLMSs,LRSs,virtual
patients,allalignedwitheachother.
CheapcomputingdeviceslikeArduinoimproveresearcheraccesstosensordatathrough
greateraccesstoabroaddevelopercommunitywithanopen-sourceattitudetodesign
recipes;reducedcostofhardwareandsoftwaredevelopment;andaneverexpandingrange
ofsensormodalities.
Bibliography
Abbey,D.(2016).LearningLockeronGithub.RetrievedJune18,2016,from
https://github.com/LearningLocker/learninglocker/releases
Arora,S.,Sevdalis,N.,Nestel,D.,Woloshynowych,M.,Darzi,A.,&Kneebone,R.(2010).The
impactofstressonsurgicalperformance:asystematicreviewoftheliterature.
Surgery,147(3),318–30,330.e1–6.http://doi.org/10.1016/j.surg.2009.10.007
Cohen,S.,Kamarck,T.,&Mermelstein,R.(1983).AGlobalMeasureofPerceivedStress.
JournalofHealthandSocialBehavior,24(4),385.http://doi.org/10.2307/2136404
Hardy,L.,Mullen,R.,&Jones,G.(1996).Knowledgeandconsciouscontrolofmotoractions
understress.BritishJournalofPsychology,87(4),621–636.
http://doi.org/10.1111/j.2044-8295.1996.tb02612.x
Kaplan,M.(2014).WhentoUseMongoDBRatherthanMySQL(orOtherRDBMS).Retrieved
June18,2016,fromhttps://dzone.com/articles/when-use-mongodb-rather-mysql
Kobielus,J.(2013).TheFourV’sofBigData.RetrievedJune18,2016,from
http://www.ibmbigdatahub.com/infographic/four-vs-big-data
Korneliusz.(2014).MongoDBvsMySQL:ComparisonBetweenRDBMSandDocument
OrientedDatabase.RetrievedJune18,2016,fromhttps://blog.udemy.com/mongodbvs-mysql/
Lalley,J.,&Miller,R.(2007).THELEARNINGPYRAMID:DOESITPOINTTEACHERSINTHE
RIGHTDIRECTION?Education,128(1),64–79.
Mei,S.(2013).WhyYouShouldNeverUseMongoDB.RetrievedJune18,2016,from
http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb
Meiselman,E.,Topps,D.,&Albersworth,C.(2016).MedbiqxAPIworkshop.InV.Strothers
&P.Greene(Eds.),MedbiqAnnualConference.Baltimore,MD.Retrievedfrom
http://www.slideshare.net/topps/medbiq-xapi-workshop2b
Miller,G.E.(1990).Theassessmentofclinicalskills/competence/performance.Academic
Medicine :JournaloftheAssociationofAmericanMedicalColleges,65(9Suppl),S63–7.
Retrievedfromhttp://www.ncbi.nlm.nih.gov/pubmed/2400509
Topps,D.,Meiselman,E.,Ellaway,R.,&Downes,A.(2016).AggregatingAmbient_Student
TrackingDataforAssessment.InOttawaConference.Perth,WA:AMEE.Retrievedfrom
http://www.slideshare.net/topps/aggregating-ambient-student-tracking-data-forassessment
Topps,D.,Meiselman,E.,&Strothers,V.(2016).MedbiqxAPIVirtualPatientProfile.
Baltimore.Retrievedfrom
http://groups.medbiq.org/medbiq/display/XIG/Profile%3A+1.+Virtual+Patients