OnlineSEIsoftwarerecommendationforFacultySenateReview,12-03-2015
The49thFacultySenateendorsedthecreationofanAdHocCommittee(alongwiththeOfficeof
theProvost)tostudyanalternativesoftwaresolutionfortherequiredstudentevaluationof
instruction,inresponsetodecliningstudentparticipationinonlineSEIandanincreaseinthe
numberofacademicdepartmentsthatwerechoosingorconsideringareturntotheuseofpaper
SEIforms.Inaddition,studyofalternativeswasrequestedbytheStudentSenate(also
concernedaboutlowstudentparticipation)andbyITS.PrimaryconcernsinITSweretheage
andcostofreplacementforhardwareneededtosupportpaperSEIforms,andthelackof
availablestafftimeformanagingandsupportingtheuseofpaperforms.
TheAdHocCommitteemembershipincludedfaculty,ADAs,andadministrativeofficers.They
evaluatedanalternativetoUWL’sownsoftwarethathadbeendevelopedtomanageSEIdata
collectioninanonlineformat,onthebasisofcapability,usability,andcost.Afteraninitial
review,theyendorsedthecreationofapilotprojectforSEIdatacollectionduringthesummer
sessionof2015,withfacultyandADAsfromseveraldepartmentsparticipating.
Thesoftwaresolutionthatwaspilotedisreferredtoas“SmartEvals”,andwasofferedforfreefor
thedurationofthepilotproject.Thisprojectconcludedattheendofthesummer,andtheresults
weresummarizedinareportfromShawnFrederickandothermembersofITS.Thatreportis
attached.TheSenateExecutiveCommitteehasdiscussedthisreportwithourCIO,Dr.Mohamed
Elhindi,andhasrecommendedthefollowingactionforconsiderationbytheFacultySenate.
• TheSenateExecutiveCommitteerecommendsthattheSmartEvalssoftwarepackage
becomethestandardsoftwarefortheonlinecollectionofSEIdatabyacademic
departmentsoftheuniversity,tobegininSpringSemesterof2016.
ThecontextofthisrecommendationincludesadesiretoeventuallyhaveallSEIdatacollection
doneviaanonlinesoftwaresolution,andtophaseouttheresource-intensivepaperSEIforms
andprocesses.AdoptionoftheSmartEvalssoftwarepackageasthestandardsoftwaredoesnot,
however,maketheuseofonlineSEIcollectionmandatoryatthistime.Whatitaccomplishes,we
believe,istoreducecostssignificantlywhileprovidingbettertoolsforonlinedatacollectionthan
thesoftwarethatiscurrentlyinuse.ItmayhelptomakeonlineSEIcollectionamorepalatable
solutionforADAs,students,andinstructors–basedontheexperiencefromthepilotprojectin
thesummerof2015.
WhileseveralmembersoftheSEIAdHocCommitteefromthespringsemester,andatleastone
memberofITSwhoplayedanimportantroleinthepilotprojectitself,arenolongeratUWL,we
expecttoreconvenethecommitteetoprovidefurtheradviceonfull-scaleimplementation.
ThecostoftheSmartEvalssoftwareisprojectedtobenomorethan$11,000peryear,including
supportservices.ITSreportsthatsupportservicesfromtheSmartEvalsprovidersweresolidand
veryresponsiveoverthecourseofthesummer.
CostofpaperSEIadministrationforthesmallnumberofdepartmentsnowusingpaperformsis
estimatedtoexceed$55,000,includingpaperandITsupport.Thosecostswillincrease
significantlyifadditionaldepartmentsreturntotheuseofpaperforms,andthereisuneasiness
abouttheabilitytomaintainoutmodedhardwareneededfortheuseofpaperformsevenif
significantdollarsareavailable.
ADAswhowereinvolvedinthepilotprojectreportthattheuseofSmartEvalssoftwareis
simplerthantheuseofUWL’scurrentonlinesoftware.
Facultywhowereinvolvedinthepilotprojectwereneutralorinfavorofchangingtothe
SmartEvalssoftware(nocompletesurveyhasbeendone,however).
StudentexperienceoftheSmartEvalssoftwareisneutralorimproved,withasignificant
improvementbeingseeninthattheSmartEvalssoftwareisfriendlyformobiledevices,andcan
thereforebeusedintheclassroomsetting.
Responseratesweresimilartothecurrentonlinesoftware,thoughinstructorswhotook
advantageoftheabilitytousemobiledevicesforstudentstorespondintheclassroomsawvery
goodresponserates.
Membersofthe“SEIAdHocCommittee”wereaslistedbelow.
• KaryAuby(MathADA)
• AndersCedergren(HEHP)
• ShawnFrederick(ITS)
• DanGrilley(Chem/Biochem,ATCCommittee)
• MaryGrattan(EconomicsADA)
• StormLarson(StudentSenate)
• AlexO'Brien(Psychology)
• KasiPeriyasamy(ComputerScience)
• BradSeebach(Biology,FacultySenate)
• NatalieSolverson(InstitutionalResearchOfficer)
• RobertHoar(AssociateViceChancellorofAcademicAffairs)
SmartEvals SEI Overview Report
Background
SmartEvalswaspilotedasareplacementtoolforcollectingSEI’stoincreasethesurveycompletion
results.ThecurrentSEIsystemisdevelopedin-houseandsupportedinITS.
•
•
•
•
SmartEvalsisawebbasedtoolwithstrongreportingfeatures.
Theapplicationwaspilotedwithsixdepartmentsrunningsummerclasses.
ThepointindividualfortheinstructorswasBradSeebachanddepartmentADAs.
ResultswerereviewedwiththeADAstogleaneaseofuseandreportingfunctionality.
Configuration
•
•
•
•
Setup
o DoesnotintegratewithD2LdirectlythroughLTI,butinstructorscanhyperlinkto
SmartEvals.
o 3maintableswereexportedfromWINGSanduploadedtoSmartEvals:
§ StudentRoster–UserIDanddemographicsonthestudents
§ Courses–Course,department,(surveystart,endandreleasedates),custom
traits(faculty/instructionalstaff)andinstructorinformation
§ Registration–CourseandstudentID
HowstudentsaresentaccesstoSmartEvals
o Studentsaresentemailswithlinkstothesurveys.
o ThroughalinkprovidedinD2L
o TheQRcodefeatureisnotabletobeleveragedduetosinglesignonsetup
Singlesignon
o SSOintegrationwithSmartEvalseliminatestheneedformaintainingseparate
passwords.
o Dailyexportoftheusers,instructorsandcoursesfromWINGSwillneedtobe
configuredtokeepthedatacurrent.
Email
o Advancecustomizableemailnotificationsexist.
o Forthesummerpilotthreeemailsweresentbutthefeaturesexisttosendmore.
o Theemailnotificationsweremanualduringthepilotbutcanbeconfiguredtobe
automated.
Response rate results
•
ActualresponseratesasreportedinSmartEvalsasbrokendownbydepartment
Not
Department Responded
Responded
Total Percentage
BIO
29
41
70
59
CHM
51
33
84
39
CYC
1
3
4
75
ECO
29
15
44
34
MTH
18
24
42
57
97
54
151
36
225
170
395
43
PSY
•
ActualresponseratesasreportedinSmartEvalsasbrokendownbycourse
Instructor
Dept
Num CourseName
Litster,M.
BIO
103 IntroBiology
101 LEC
3
12 25%
Galbraith,A.(team)
BIO
306 Genetics
101 LEC
5
13 38%
Osmundson,T.(team)
BIO
101 LEC
5
13 38%
Seebach,B.
BIO
306 Genetics
HumanAnatomy&
312 PhysiologyI
101 LEC
33
45 73%
Bryan,J.
CHM
103 GeneralChemI
201 LEC
17
42 40%
Carmosini,N.
CHM
103 GeneralChemI
21L LAB
8
21 38%
Kirsch,J.
CHM
103 GeneralChemI
22L LAB
8
21 38%
Tobin,C.
CYC
450 InternChild/Youth
413 FLD
3
4 75%
Nunley,J.
ECO
110 Microeco&PubPol
431 LEC
7
20 35%
BabakhaniTeimouri,S.
ECO
120 GlobalMacroecon
421 LEC
8
24 33%
Baumann,D.
MTH
145 ElemStatistics
102 LEC
20
23 87%
Allen,R.
MTH
145 ElemStatistics
203 LEC
4
19 21%
Basten,B.
PSY
100 GeneralPsychology
411 LEC
3
24 12%
Tobin,C.
PSY
421 LEC
10
26 38%
Tobin,C.
PSY
411 FLD
2
9 22%
Basten,B.
PSY
204 AbnormalPsychology
VolunteerExperiencein
309 Psyc
ResearchMethodsPsy
321 Minors
411 LEC
12
23 52%
McKelley,R.
PSY
334 HealthPsychology
411 LEC
8
21 38%
OBrien,A.
PSY
432 LearningandMemory
421 LEC
8
19 42%
Tobin,C.
PSY
450 FieldworkExperiencePsy
411 FLD
3
6 50%
Tobin,C.
PSY
450 FieldworkExperiencePsy
412 FLD
3
5 60%
Quartiroli,A.
PSY
451 PsychMeasurement
411 LEC
5
18 28%
•
•
Sec
Type N
Tot %
2of21classesmeetorexceededthegoalof75%responserate.
Whenthereportswerereviewed,eachsuccessiveemailsentgenerated10%moreresponse.
•
Marketingthesurveyseemstobethekeyfactorinthehigherresponserates.
Reports
•
•
Whattypeofinformationisincludedonthereports
o FractionalmedianasdefinedbyUWLisincluded.
o Listofcommentsareincludedinthesurveyresults.
o Graphicrepresentationsareoptionaltoincludeforquickassessment.
o Morecomprehensivetoolsforreportingresultsareavailableusingthecustomreports.
Thereportscanbefilteredby:
§ Evaluationperiod
§ Individualorgroupclasssummary
§ Classsections
§ Departments
§ Instructors,individualsorgroup
§ CustomfeatureofFacultyorinstructionalstaff
Whatthereportsdonotinclude
o Thecalculatedfractionalmedianforquestions2-6wouldneedtobedevelopedbythe
vendoroutsidethescopeofthepilot.
o DoesnotintegratewithTAI’s(TeachingAssignmentInformation).However,the
SmartEvalsreportscanbegiveninadditiontowhatisprovidedthroughWINGS.
Summary
SmartEvalsdidnotmeettheintendedgoalof75%responserate.Somefactorsthatmighthaveaffected
theresultswerethattheQRcodeswerenotabletobeusedandtheemailfrequencywaslimitedto3
total.SmartEvalshasanumberoffeaturesthattheADAslikedduringthesurveyresultsreview,the
abilitytogeneratedetailedreportseasilyandtocustomizethereportstoaddressdifferentcriteria
requestwasemphasized.Theoptionexistforexportingthesurveyresultstomanydifferentformats,
mainlyfocusingonexcelorPDF.SmartEvalshasadvancereportingdemographicfeaturesthatwerenot
usedinthepilotbutcouldbeexploredinfutureimplementations.Duringthepilotthesetupwas
controlledandconfiguredtoalargepartbyIT,forproductionwewouldhavetoconfirmthattheADAs
wereabletoconfiguretheclasses.SmartEvalscoversthesamebasicinformationasthecurrentSEI
collectionsystemwithmoreadvancereportingfeatures.Responseratesaresimilartothecurrent
systemwhichsuggestthatthemarketingofthesurveysneedstobeaddressed.