Moving into the mainstream: Researching the institutional
introduction of EVS, from the realm of the enthusiasts to
supporting the later adopters
Marija Cubric
Business School
www.herts.ac.uk
Amanda Jefferies
School of Computer Science
Contents
Background
Literature review
Methodology
Preliminary findings
Focus on staff views on technology adoption and use of EVS
Conclusions
Background
Major institutional investment by the university in 2010-11 (Assessment and Feedback project)
• 3,845 students form 8 academic schools issued with a personal handset from Jan 2011
JISC EEVS project (stand B) , alongside iTEAM Strand A , started in September 2011
• EEVS = Evaluating Electronic Voting Systems for Enhancing Student Experience
EEVS Objectives
• To provide an up-to-date view of the student and staff experiences of EVS
• To identify a set of critical success factors for introducing and maintaining the use of EVS in
support of an institutional assessment and feedback strategy.
• Focus on large-scale deployment and differences across multiple disciplines.
Literature Review: Use of EVS in HE
Survey of literature on the introduction of EVS in HE
•
•
300+ papers on published between 1998 and 2011
A wide variety of writers :
– Brown, Davis, Draper, d’Inverno, Kennedy, Nichol, Oliver, White et al
– Our own early adopters :- Lorimer, Hilliard, Thornton and Willis (University of
Hertfordshire)
Most authors reported on local use in classes:
• to engage students,
• to encourage interactivity,
• to support constructivist approaches to teaching and learning in line with e.g. Chickering and
Gamson’s guidelines.
Lack of literature on institutional deployment, with the exception of:
• Twetten et al (2007) Successful Clicker Standardisation, Educause Quarterly 4
– Experience from EVS standardisation in four North American universities
Methodology: Data
Exploratory pragmatic approach:
• Gaining understanding of the meanings from various stakeholders
• Focus on practical applied research, integrating different perspectives to help interpreting data
(Sounders et al 2009)
Mixed-method data collection:
• Qualitative data: students’ blogs, staff interviews, staff reports
• Quantitative data: students’ questionnaire, staff questionnaire
Time horizon: Mainly cross-sectional (a ‘snapshot’ taken in 2011/12) with some elements of longitudinal
study (students’ blogs over a period of 2-4 weeks).
Sampling:
• Self-selection (voluntary student participants for questionnaires)
• Self-selection & purposive (voluntary student participants for blogs; further selection made to enable
balanced representation across different schools)
• Purposive (heterogeneous): staff involved with or leading EVS adoption in different Schools
Methodology: Students’ survey design
Survey launched in February
2012
Includes 27 questions spread
across 9 sections
Total number of responses
N= 590 (across 11 different
Schools)
Response rate: 14.4%.
(based on 4093 EVS handsets
registered in June 2012 )
Section
Source
Background
Standard demographic
Information
questions
Usage
(Venkatesh et al, 2003
Ease of use
Bangor et al , 2008
Impact on Learning
Draper & Brown 2002
Performance
Venkatesh et al, 2003
Expectancy
Engagement
JISC Evaluation guidelines
Satisfaction
JISC Evaluation guidelines
Operational procedures
iTEAM questions
and funding
Free text comments
Methodology: Staff survey design
Survey launched in June 2012
Section
Includes 18 questions spread
across 5 sections
Background
Standard demographic questions
Information
Technology adoption (Venkatesh et al, 2003)
(Venkatesh et al, 2003) + iTEAM
Usage
questions
Effect on teaching
JISC Evaluation guidelines
practice
Impact on Learning Draper & Brown 2002
Operational
procedures and
iTEAM questions
funding
Free text comments
Total number of responses
N= 88 (across 13 different
Schools)
Response rate: 5.9%.
(based on 1500 academic
staff registered on staffq list
in June 2012)
Source
Methodology: Students’ blogs: Participants
Student blogs were completed in a variety of ways according to individual preferences :
• Word Files (46), online blog entries (14), Short video clips (11), Audio files (1) and
• submitted via a restricted access group area on StudyNet (UH MLE)
65 students registered for the web/ written blog of whom 27 were chosen to ensure a spread of gender
and programme. One failed to complete their blog
7 Academic Schools: Business (6), Computer Science (7), Humanities (6), Law (4), Education (1), Sports
management (1), Mathematics (1)
Equal male and female numbers applied but ratios varied according to subjects 15M:11F
All the bloggers had previously used EVS since at least January 2011
The reflection period varied between 2 and 4 weeks in the period Nov 11- Feb 12
Methodology: Student blogs:
Original and Emerging Categories and Themes
All Categories
Pedagogy
Sub-categories (Themes)
Examples of use ,Interesting lectures, Subject suitability, Subject-specific EVS use,
Improvements in use, Potential use, Learning
Total
292
Formative use, Inadequate use, Misaligned expectations, Variety of assessment, Edutainment
and more …
Usage
Technology
Previous experience with EVS ,Use by modules and levels ,Frequency of use
Type of handset ,Technical issues ,EVS functions ,
Changing channels
78
99
Institut’l benefits
Attendance ,Institutional motivation
47
Expectations
Initial thoughts ,Institutional use of data
37
Effort Expectancy
Ease of use, Accessibility
40
Facilitating conditions
Training ,Handsets distribution
Technical competence of tutors, Tutor's experience with EVS use/question writing
56
Institutional
deployment
Handset ownership
Social aspects
Anonymity
28
Student experience
Satisfaction and Dissatisfaction
21
Personal emotions
Anxiety, and extra pressure, privacy
7
Handset cost, Inconsistent application, Too ambitious implementation
32
Methodology: Staff interviews: Participants
and Developing Categories and Themes
Semi-structured
interviews with
12 staff members incl:
• Local School
Champions (2)
• Associate Heads
and Heads of
School (3)
• Lecturers (6)
• The iTEAM Project
Director (1)
Original
Categories
Sub-categories (Themes)
Technology
Adoption
Drivers for change,
Institutional Support for technology adoption,
Positive factors of EVS
Negative factors of technology adoption
Negative factors of EVS usage
Embedding EVS successfully
Training & Support at institutional level
Support at local level
Champions for EVS
Benefits of Training
Personal
Competence
Changing styles Use of MCQs for learning and teaching
Developing Question Banks
of Pedagogy
Benefits of EVS for Student support
Preliminary Findings : Summary of students’
views (survey and blogs)
Usability or ease of use of the EVS was rated very highly by
students
Time given for answering questions is not always
sufficient
Very positive impact on students’ learning and satisfaction:
• Responding to questions makes me think more about the
course material (80%+)
• EVS provides them with an immediate check of their
understanding (80%+)
• Viewing responses give them an idea of how they are doing in
relation to the rest of the class (80%+)
Formative use of EVS has been seen as adding a
value to students’ learning experience, however
summative use has in some cases created “an
unfortunate tension between the administrators,
the tutors and the students” (MC12)
Less positive impact on perceived performance and engagement :
• use of EVS will increase their chances of getting a good grade
(49.9%)
• Spend more time discussing the subject with colleagues
(47.5%)
• Thinks about the subject outside of the classroom (34%)
• Ask more questions in classes that use EVS (21%)
All students find the EVS very easy to use, with
little or no training; however, accessibility might
be an issue for students with sight or dexterity
impairments. Majority agree that EVS is a great
tool for answering QUESTIONS, but inadequate as
a TEST tool
Preliminary Findings : Staff Survey
There is an overall (UH-wide) highly positive attitude to technology
(EVS) acceptance with 83% of surveyed staff intending to use EVS in
the next 6-12 months.
• A more positive attitude in schools from Social Sciences and Arts
subject area than in other subject groups
There is an overall (UH-wide) high agreement on handing out EVS
for free upon registration and disagreement that the use of EVS has
led to immediate operational efficiencies
Preliminary Findings : Staff interviews
The Head of School
‘Our biggest challenge was getting staff on board with it. I
would think when I started, 25% of staff were very
enthusiastic and 75% were not, now we’re probably
running at 50/50.
Everybody who’s used EVS and got it to work comes back
and says it’s great and they’re going to use it again… the
technology has got to be easy so that people can get in the
swing of it’
Head of School interview June 2012
Preliminary Findings : Staff interviews
The Lecturers
‘Once it’s embedded in
the workload for staff
then that should be
greatly reduced in terms
of marking, writing
assignments, it’s really
refreshing the bank of
multiple choice questions
[they’ve created.]’
…it was so obviously simple
to generate questions and I
think that has always been
the main barrier in people’s
minds, that because it’s
something new and that it’s
a technology that it has to
be difficult as well
Conclusions
• High student enthusiasm for using EVS in class especially for
formative and low-stakes summative assessments
• Plenty of top-down support and leadership and local support is
important to embed the technology
• Academics using EVS are not yet seeing all the expected
efficiencies in time saved that they had hoped for.
• BUT 93% of those responding to the survey and using it for the
first time in 2011 said they plan to use it in the next 6 -12 months
• Patterns of previous technology adoption at UH are mirrored –
– The example of the MLE saw early student enthusiasm but
academics took more time to feel fully at ease with the change
in practice and pedagogy.
References
•
Bangor, A., Kortum, P., & Miller, J.A. (2008). The System Usability Scale (SUS): An Empirical Evaluation, International Journal of Human-Computer Interaction, 24(6).
•
Chickering and Gamson http://www.uis.edu/liberalstudies/students/documents/sevenprinciples.pdf
•
D’Inverno, R., Davis, H., & White S. (2003), Using a personal response system for promoting student interaction, Teaching Mathematics and its Applications, Vol. 22 (No.
4): 163-169.
•
Draper, S. W., & Brown, M. I.(2002).Use of the PRS handsets at Glasgow University, Interim Evaluation Report: March 2002
http://www.psy.gla.ac.uk/~steve/evs/interim.html accessed April 2012
•
Draper, S. W., & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20(2), 81-94.
•
JISC (2004) http://www.jisc.ac.uk/media/documents/publications/effectivepracticeelearning.pdf
•
Kanter, R.M., Stein, B.A. & Jick, T.D. (1992) The Challenge of Organizational Change. Free Press. New York
•
Kennedy, G. E., & Cutts, Q. I. (2005). The association between students' use of an electronic voting system and their learning outcomes. Journal of Computer Assisted
Learning, 21(4), 260-268
•
Lorimer, J., & Hilliard, A. (2009). Use of a Electronic Voting System (EVS) to Facilitate Teaching and Assessment of Decision Making Skills in Undergraduate Radiography
Education. Paper presented at the Proceedings of 8th European Conference on e-Learning, Bari, Italy
•
Moore, G. A. (1991). Crossing the Chasm. New York: Harper Business.
•
Nicol, D., & Draper, S. (June 2009). A blueprint for transformational organisational change in higher education: REAP as a case study. In J. T. Mayes (Ed.), Transforming
Higher Education through Technology-Enhanced Learning.
•
Oliver, M. (2006). New pedagogies for e-learning? ALT-J: Research in Learning Technology, 14(2), 133 - 134.
•
Robins, K., (2011) EVS in the Business School, University of Hertfordshire Internal Report
•
Thornton, H. A. (2009). Undergraduate Physiotherapy students’ choice and use of technology in undertaking collaborative tasks. Open University, UK, Milton Keynes
•
Twetten, J., Smith, M.K., Julius, J. and Murphy-Boyer, L. (2007) Successful Clicker Standardization EDUCAUSE QUARTERLY • Number 4 2007
•
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. B. (2003). User acceptance of information technology: toward a unified view. MIS Quarterly, 27(3), 425–478.
•
Willis, J. (2009) Using EVS in the School of Life Sciences, University of Hertfordshire Internal Report