Evaluation and Replacement of Individual
Development and Educational Assessment
(IDEA)
Stephen Burd (burd@unm.edu)
Associate Professor, ASM
Academic Technology Liaison
Presentation copies available online
http://averia.unm.edu
Last revised: 7/17/2016 3:49 AM
Project Context
In summer 2012, Associate Provost (Greg Heileman)
charged the Academic Technology Liaison (Stephen Burd)
to identify and evaluate alternative tools for student
assessment of courses and instructors
Rationale:
High administrative complexity of current system
Difficulty in gathering/using survey responses/results for further
analysis (e.g., data analytics and text mining)
Concerns about usefulness of results in promotion and tenure
evaluation
Faculty dissatisfaction with current system
A working group was formed with most faculty members
drawn from the Faculty Senate Teaching Enhancement
and IT Use Committees
http://averia.unm.edu/IdeaNextStep
Working Group Members
Faculty
Other
Stephen Burd (ASM)
Robert Busch (Chemical & Nuclear
Engineering)
Moira Gerety (Deputy Chief
Information Officer)
Greg Heileman (Associate Provost for
Curriculum)
Kevin Comerford (Library)
Nick Flor (ASM)
Grace Liu (ASUNM)
Kristopher Goodrich (Counselor
Education)
Kris Miranda (GPSA)
Chris Holden (Honors)
Amy Neel (Speech & Hearing)
Caleb Richardson (History)
Mary Margaret Rogers (ASM)
Julie Sykes (Spanish & Portuguese)
Goals for IDEA Replacement (IDEA-R)
Increase use of and usability of student feedback
on courses/instructors for formative and
summative purposes
Adopt a modern tool with:
Greater flexibility for faculty, departments, and
programs
Online and mobile survey capabilities
Improved reporting
Support for analytics
These goals drove the RFP authoring process
Timeline
Fall 2012 – Working group examines faculty technology survey results and available
products – determines that a replacement for IDEA is warranted
Spring/Summer 2013 – Working group examines available alternatives and sample
RFPs in detail – develops/releases RFP
Fall 2013 - RFP responses close in October, preliminary evaluation begins
Spring 2014 – Detailed evaluation of RFP responses, top responses identified,
vendors demoed in early May:
ConnectEDU (CouseEval)
EvaluationKit – used at CNM and NMSU
eXplorance (Blue)
June 2014 – AVP Heileman reviews choices and feedback, chooses EvaluationKIT,
working group concurs unanimously
July-Sept 2014 – Acceptance (sandbox) testing is successful
Sept-Oct 2014 – Contract negotiated – Provost approves purchase
Oct–Dec 2014 - Steering committee formed, pilot testing, initial discussion of related
policies with Faculty Senate
Spring 2015 – Evaluate pilot results and make adjustments, phase 1 rollout to 3350% of UNM, finalize related policies
Summer 2015 – Full switchover to EvaluationKIT
Summary of Finalist Evaluations
eXplorance (Blue) – “The Lexus”
Tops in functionality/features
Much more expensive than the other two (± $400K)
EvaluationKit – “The Hyundai”
Acceptable in functionality/features
Reasonable cost (< $40K)
Some reservations about:
Ease-of-use
Usability beyond summative end-of-semester evaluations
ConnectEDU
Barely acceptable in functionality/features
Significant concerns about viability of vendor, adequate
resources, and strategic direction for future product
development
EvaluationKIT Selection Reasons
License and operational cost a bit less than IDEA
Positive feedback from CNM and NMSU
Satisfies “must-have” requirements
Moves us firmly into the 21st Century
Gets us out of the paper shuffling business
Extra features of eXplorance are unlikely to be
used in the near term
Alternative tools exist for ad-hoc instructor-initiated
surveys (e.g., UNM Learn, Opinio, paper, …)
Key EvaluationKit Features
Survey structure similar to old ICES system
Develop a UNM question bank and/or “roll-your own” questions
A survey can “layer” questions from multiple organizational levels
No explicit tie to learning objectives or inter-institutional norms
Best use is mid-semester and end-of-semester evaluations – not
well-suited to ad-hoc instructor-initiated surveys
Fully online system
Hosted on vendor servers – no local installation option
Survey definition and administration via browser-based
application
Students complete surveys via browser or cellphone app
Reports generated in PDF/Excel and viewed online, delivered via
email, or downloaded
Surveys/results can be extracted for downstream analytics
To Where From Here?
Fall 2014
Start policy discussion with the Faculty Senate
Plan/execute first small pilot for Fall end-of-semester evaluations
Participants ASM, Architecture, Public Administration, UNM Gallup
Experiment with:
Centralized and distributed administration
Survey content
Survey open/close dates – response rate impact
Email communication with students – response rate impact
Other communication with students – response rate impact
Spring 2015
Evaluate first pilot results and plan phase 1 roll-out
Who will participate in this roll-out?
Develop training materials
Plan summer/fall roll-out
Summer/Fall 2015
Turn off IDEA
Roll-out EvaluationKIT across UNM
Policy Issues for Faculty Senate Consideration
Administration
How will control over survey content and administration be distributed
among academic affairs, schools & departments, faculty, central IT
services?
Tool specificity
Should use of a UNM-approved tool be required?
Survey content requirements
Will UNM adopt a set of standard questions included in all surveys? If
so, what are they?
Will UNM populate an institutional question bank from which questions
can be chosen and/or enable schools, departments, and instructors to
create their own?
Confidentiality of survey respondents
Is existing language too weak, about right, not strong enough?
Distribution and/or confidentiality of survey data and reporting
Who gets to see what data/reports and under what conditions?
Do students or the public get to see any of it?
EvaluationKIT Mobile Interface Examples
EvaluationKIT
Browser-Based Interface Example
EvaluationKIT
Instructor/Course Report Example
Report Example - Continued