Hearing the Online Student Voice: Addressing student perceptions of technology
enhanced learning adoption
Emily Webb, Rod Cullen, Irfan Mulla, Hannah Palin, Susan Gregory
and Osman Javaid
Manchester Metropolitan University, li@mmu.ac.uk
ABSTRACT
Manchester Metropolitan University (MMU) has invested significantly
in developing technological infrastructures to support learning, teaching and assessment. At the
core of this infrastructure is a Moodle Virtual Learning Environment that integrates tightly with
the Student Records System (SRS) and other institutional systems including timetabling,
MMUtube (audio/video service), Library Systems (Talis Aspire), and coursework receipting in a
way that wraps the institution around the individual learner.
These developments have been informed by ongoing consultation with students including an
annual Internal Student Survey (ISS) which enables students to provide quantitative and
qualitative feedback on their learning, teaching and assessment experience on each unit within
their programme of study.
Students in the 2014 /15 Internal Student Survey posted 47,800 free text comments. Filtering
these using a set of key words extracted 2072 comments related to the student experience of
Moodle and other aspects of technology enhanced learning, teaching and assessment. This
provided a data-set of 746 comments relating to “Best” aspects of programmes/units and 1326
comments relating to aspects of programmes/units that students would like to see improved.
A detailed thematic analysis of these data has been undertaken. Eighteen themes, arranged
into six categories, emerged from the analysis of comments relating to the “Best” aspects of
programmes/units. Twenty-five themes, arranged in seven categories, emerged from the
analysis of aspects considered to be “in need of improvement”. We have subsequently been
able to trace both “best” and “improvement” comments back to individual Moodle unit areas in
an attempt to characterize what these “look like” from the student perspective. This paper will
explore the characteristics of “best” and “improve” themes and the strategic approaches we
have subsequently adopted across the whole institution to build on the best and to support
improvements that will enable academics and students to get the best out of Moodle.
KEYWORDS
Student Voice, Technology Enhanced Learning, VLE, Evidence-based Action
SOLSTICE Conference 2015, Edge Hill University
1
Introduction
Many factors influence the successful adoption of Technology Enhanced Learning (TEL) within
the higher education environment. These include student and staff digital literacy, support
infrastructure and organisational culture. (Reed and Watmough ,2015) One compelling factor is
the student voice, which can aid self-improvement, and in turn influence university advancement
(Alderman et al, 2012). Listening carefully to student accounts of positive and negative TEL
experiences can provide a practical agenda for improving teaching, teaching and assessment
practice. In this respect, taking account of student practice and testimony can feed into
evaluation, strategic planning, university policy and improvement. (Langhan et al, 2015)
Context
Manchester Metropolitan University (MMU) has invested significantly in developing
technological infrastructures to support learning, teaching and assessment. At the core of this
infrastructure is Moodle, a Virtual Learning Environment (VLE) that integrates tightly with the
Student Records System (SRS) and other institutional systems including timetabling, MMUtube
(audio/video service), Library Systems (Talis Aspire), and Coursework Receipting in a way that
‘wraps the institution around the individual learner (Stubbs, 2014 p.1).
These developments have been informed by ongoing consultation with students including a biannual Internal Student Survey (ISS). The ISS is MMU’s own “in-house” institutional survey on
taught, campus-based programmes. All currently enrolled students are invited to
provide quantitative and qualitative feedback on their learning, teaching and assessment
experience, on each unit within their programme of study. Student consultation and opinion is
an essential component of the evidence base for programme approval, review and modification
processes. MMU states that:
“The evaluation of student opinion and appropriate response to the results is a key
indicator in the University's processes for the assurance and enhancement of academic
quality and is a required element of the evidence base for the Continuous Monitoring
and Improvement process and for periodic review.”
http://www.mmu.ac.uk/academic/casqe/experience/student-voice.php
MMU works closely with the Student’s Union to evaluate and respond to student opinion. The
Continuous Monitoring and Improvement process aims to:
“… …support the maintenance of standards, to assure the consistency of learning
opportunities and to enhance the quality of the learning experience for students by
continually reviewing provision, identifying areas for improvement and taking appropriate
and timely actions.”
http://www.mmu.ac.uk/academic/casqe/experience/monitoring-improvement.php
Whilst providing opportunities for student voice and feedback has become established practice
within Higher Education Institutions, Alderman et al. (2012) state that there is little evidence to
demonstrate how feedback and student voice feeds into action and how the outcomes of these
actions are fed back to the students.
SOLSTICE Conference 2015, Edge Hill University
2
Harvey (2011) proposes that a feedback action cycle (Figure 1) provides a suitable framework
that ensures that student feedback feeds into action and consequently those actions are fed
back to the students. This paper seeks to demonstrate how effective mechanisms for collecting
feedback lead to tangible actions to enhance the use of the VLE and other learning technologies
at MMU.
Figure 1 - Feedback Action Cycle
Phase 1 – Data Collection and Analysis
Extensive qualitative data was collected through two free text questions in the 2014/15 ISS.
These invited students to respond to the following statements:
The best thing about this unit is:
In need of improvement within this unit is:
This resulted in students leaving 47,800 comments.
These comments were subsequently filtered using 24 key words (Moodle; wifi; wireless; print;
pc ; pcs; mac; macs; software; drop in; mobile; twitter; phone; video; ipad; tablet; facebook;
podcast; email; matlab; mymmu; app; mmutube; padlet; it zone). This extracted 2072
comments related to the student experience of the institutional VLE and other aspects of
technology enhanced learning, teaching and assessment. This provided a data-set of 746
SOLSTICE Conference 2015, Edge Hill University
3
comments relating to “Best” aspects of programmes/units and 1326 comments relating to
aspects of programmes/units that students would like to see improved.
A variation of Richie and Spencer’s (1994) ‘Framework’ approach was employed to analyse the
comments. This involved a systematic process with five key, interconnected stages:
1. Familiarization - Two members of the learning innovation team read though the text
responses independently and began to pick out repeating themes.
2. Identifying a thematic framework - The same two team members compared and
contrasted notes and agreed a common set of coded themes.
3. Indexing – Each of MMU’s eight Faculties has a dedicated Learning Innovation Team
member who indexed (coded) the text response made by students in their Faculty
against the agreed thematic framework.
4. Charting - The team charted the frequency of themes within the data set for the
institution as a whole as well as for each individual faculty. In this respect the data were
used to produces graphs and tables of the frequency of the coded themes.
5. Mapping and interpreting – The team worked to together to further explore the
relationships between themes.
Results
Best Themes
Figure 2 provides an institutional overview of the frequency of “best” themes.
SOLSTICE Conference 2015, Edge Hill University
4
Figure 2 - Institutional Overview of best themes
Highest Scoring Best Themes relating to VLE
Theme
Content well organised/high quality
Provision/use of audio/video materials
Effective communications
Moodle up to date/advanced material
Result
183
166
127
122
Content well organised/high quality
This was the highest scoring theme, with Moodle being used effectively to organise high quality
learning materials.
“The wide resources taught e.g. solving questions with software packages as well as by
hand. The lectures and tutorials are organised well and a good way to learn the content
in the course.”
SOLSTICE Conference 2015, Edge Hill University
5
Provision/use of audio/video materials
In some cases the use of Moodle by staff has advanced from using the VLE as a digital
repository towards developing a more interactive learning environment. Students rated highly
the interactivity of audio and video materials. The range of resources appeals to a variety of
learning styles and promotes engagement.
“The organisation of the unit is one of the best things about this unit. The use of
podcasts and video lectures alongside traditional lectures and seminars were a good
combination. The content was not too heavy and was all digestible in appropriate
chunks.”
Effective communications
The appropriate use of Announcements within Moodle provided clear communication for
students. When an Announcement is posted the student receives the information through two
channels, the Moodle area and via student email. This prevents students from being absent
from email lists that have been generated from various sources, and encourages the students to
take responsibility to check their Moodle units on a regular basis.
“The communication between groups of the course is improving and suggestions have
been implemented to help share knowledge from the course tutors (Facebook/Moodle
groups).”
Moodle up to date/advanced material
Staff are taking an active part in keeping Moodle up to date and the students appreciated this.
The availability of current content provided a rich learning resource for students. The presence
of advanced learning materials was advantageous to the student, as prior to the lecture they
could print out the material and then annotate with their own notes.
“The quality of teaching was amazing and with all the lecture Powerpoints being on
Moodle in advanced I was able to start my essay in advanced as I had basic notes
already on it.”
However, when analysing the “to improve” comments there is evidence to suggest that
PowerPoint slides alone are not advantageous, with 47 comments indicating that the
inappropriate use of PowerPoint is an area for development. Students express that they require
further support alongside lecture presentations with additional resources such as accompanying
explanatory video/audio files.
Improve Themes
Figure 3 provides an institutional overview of the frequency of “improve” themes.
SOLSTICE Conference 2015, Edge Hill University
6
Figure 3 – Institutional overview improve
Highest Scoring Improvement Themes in Moodle
Theme
Organisation and quality of resources in
Moodle
Communication (clearer information/faster
responses)
Upload teaching materials to Moodle
Advance availability of teaching materials
via Moodle
Result
181
155
138
105
Organisation and quality of resources in Moodle
Having an organised Moodle area with good quality resources appears as the highest rated
theme in the “Best” category. In comparison, the highest rated theme in the needs improving
category were Moodle areas that were disorganised and contained poor quality resources. At
first glance it seems that the students are being contradictory about the same units. As such,
SOLSTICE Conference 2015, Edge Hill University
7
this phenomena infers that well organised Moodle units and high quality units are praised by
students and units that are disorganised and contain poor quality resources are criticised. A
further explanation for the theme appearing in both categories is that students like consistency
across units. Students could study one unit that is well organised and contains high quality
resources, and another unit that is poorly populated with unclear and disorganised content,
making for an inconsistent student experience. This discourse can be due to varying staff
teaching styles and different levels of engagement with Moodle.
‘’This unit is very unorganised and I feel like the tasks that we are completing in the class
do not relate to the exam that we had to complete. The slides that were uploaded to
Moodle were not in any specific order and it was very confusing to revise from for the
exam. Last year in product awareness it was very informative and exciting this year it
just feels like we aren’t progressing.’’
Communication (clearer information/faster responses)
Students do not appreciate instances where staff are using Announcements inappropriately.
This can be through unclear information, untimely or too high a quantity of messages which
could promote student disengagement.
‘’Course organisers need to develop effective communication skills. Eg last week
information about a room change was given by email 30 minutes before the lecture. As it
turned out the original information was incorrect. The staff are too laid back and give
very little information and what they do give is often contradictory. Information is often
hidden in obscure places in noodle or is it muddle.’’
In addition to receiving untimely messages, students require timely responses to their own
messages. Again, this relates back to consistency. If academic staff explain to students all
emails will be answered within 48 hours clear expectations are set. However if a member of
staff initially responds to emails immediately then over a period time finds that they are unable
to respond until 24 hours later – to the student this indicates that emails are not being
responded to in a timely manner. A quick solution to this issue would be for the tutors on the
programme to agree a set period of time within which students can expect a response. This in
turn provides a consistent approach to how communications are managed.
Upload teaching materials to Moodle
Students like to see all of their units in Moodle contain teaching materials, providing a consistent
Moodle experience.
“Would be nice if X would make his lecture slides available on Moodle for the sake of
note taking. Often what is being said is more important than the notes, but the notes are
also needed.”
Advance availability of teaching materials via Moodle
There is an appreciable desire by students to receive teaching materials in advance of the
class. From practical experience, there is a reluctance by some staff to post material in
SOLSTICE Conference 2015, Edge Hill University
8
advance in case students will not attend the class, however as the comment below indicates
this may not be the case.
“X teaches really well as she explains things thoroughly and although the slides are up
on Moodle. I feel like I would miss out if I didn’t attend.”
Automatically generated content
MMU Moodle is highly integrated with other institutional teaching, learning and assessment
infrastructure and a bespoke block on the right hand of all Moodle unit areas contains
automatically generated personalised content. This includes library reading lists and
assessment due dates and moderated marks. The rationale for automatic content is
to address certain hygiene factors that are known to contribute to dissatisfaction amongst
students (Stubbs,2014 and Reed and Watmough, 2015). Hygiene factors include: having
incomplete reading lists and not knowing assignment hand in dates. However, academic
colleagues are responsible for maintaining all other content in Moodle and the ISS comments
have highlighted that there is inconsistency between the automatically generated content and
the content developed by academic staff. For example, some colleagues chose to add
alternative reading lists to Moodle rather than updating things via the library system. Whilst the
comments relating to this issue are quite low (14 comments in total) it is considered that the
learning innovation team need to address this through training and staff development
provision. Furthermore, some colleagues unilaterally decide to change assessment deadlines
and make this known via Moodle content tools or announcements rather than changing details
via the correct central systems leading to different dates in the automated block and tutor
content. This inconsistent approach re-instates the hygiene factors that the automatically
generated content have removed, and could have a significant negative impact on student
satisfaction.
‘’Although the course is organised in terms of timetabling projects deadlines etc there is often
contradiction on what is required in terms of briefs and outcomes’’
Students as partners
In addition to analysing the emphasis students placed on what were the “best” elements of the
unit and what required improvement it became clear that students were also able to offer
valuable insights in to how practice might be improved:
“Also the seminars are fun but i never feel like the notes are collated/organised for my
personal reflection when they upload our notes to things such as padlet. Perhaps half
the time should be interactive/group work and the other half could be a presentation that
has all the information on and if not shown in the lecture could be uploaded to moodle
for personal reflection’’
‘’I would prefer to have watched videos or listened to podcasts at home and discussed
them in class instead of spending so long doing so in class.”
Often concerns in Moodle are a reflection of face-to-face concerns. This highlights the intrinsic
link between face-to-face and digital delivery of a course.
SOLSTICE Conference 2015, Edge Hill University
9
“Feel lost in the course. No flow throughout the weeks so feel unsure as to where we are
and how everything fits in together. Reading for the week or notes from the lecture are
not on Moodle so you can’t access them if you want to refer to them later on. As far as I
can see little communication between lecturer and seminar tutor so the topics are
different in each difficult to get a solid understanding.’’
‘’The content is very interesting and is delivered very well through X and Y during the
lecture. The tutorials are very helpful to build upon the weeks lecture material and
moodle is always very organised with tutorial questions and readings which keeps me
involved and makes sure you are reading relevant information.”
Phase 2 - Action
Tracing both “best” and “to improve” comments back to individual Moodle unit areas enabled the
team to attempt to characterize what these “look like” from the student perspective. This
realistic picture enables the identification of units that would benefit from increased support and
training.
Once the data had been analysed the Learning Innovation team agreed that each Technology
Enhanced Learning Advisor (TELA) would present the findings from the thematic analysis to the
faculty that they supported and then begin to work with colleagues within the faculty to develop
an action plan. The team were keen to work in partnership with the faculties in terms of how
best to develop an action plan. Below is a summary of how some faculties are planning to
respond to the outcomes of the thematic analysis:
Faculty of Humanities, Languages and Social Sciences
The outcomes form the ISS data have been presented to the faculty executive group. Staff
development sessions have been developed in conjunction with the learning and teaching
adviser from each programme. The sessions that are being delivered across the faculty are as
follows:
Awareness raising session - ‘The Student View: What We Want in Moodle’ the session will
feedback to staff the outcomes from the ISS thematic analysis.
TEL Coffee Club - informal staff development sessions that demonstrate how to use the
technologies that have been emphasised by students as being the best aspect of their learning
experience.
In addition to the staff development sessions, Moodle templates are being developed for each
area to ensure consistency across units.
Faculty of Health, Psychology and Social Care
The Dean of the faculty and the Technology Enhanced Learning group met to discuss the
outcomes from the ISS data. The group proposed two distinct strands for staff development.
Moodle Essentials – Programme teams are to work with the faculty e-learning team to develop a
Moodle essentials template to ensure an equitable and good quality student experience within
SOLSTICE Conference 2015, Edge Hill University
10
Moodle. Unit leaders are responsible for implementing the Moodle essential template with
support from the e-learning team via online and face-to-face drop-ins.
Try it on sessions – The ISS comments highlighted that students valued a range of good
practice across the faculty. The try it on sessions will provide staff with the opportunity to share
good practice and have a go at using different types of technology and teaching techniques to
promote student engagement. A member of academic staff, and a member of the e-learning
team will facilitate the sessions and they will last 45 minutes. The sessions will run at different
times and days so all staff have the opportunity to attend.
Faculty of Business and Law
Prior to the ISS extensive work had been undertaken within the business school to audit all
Moodle areas to ensure they met the minimum threshold standards which are that each Moodle
area contains: a reading list, unit handbook, unit content and assignment hand in dates. The
ISS comments have been cross-referenced with units involved in the audit to develop a unit-byunit programme of staff development.
Faculty of Science and Engineering
The ISS thematic analysis complements existing work that is already underway within the
faculty. Prior to the ISS analysis, an audit of Moodle areas was undertaken in one department
to ensure that the minimum threshold standards for each Moodle unit had been met.
Programme support tutors who carried out the audit are now using the outcomes from the audit
and ISS comments to develop a departmental Moodle template that will be placed in each unit.
The technology enhanced learning group are meeting to develop a programme of staff
development to share and promote the good practice identified by students via the ISS.
Phase 3 – Feeding back to the students
A continual dialogue with students is essential to close the feedback-action loop. MMU already
ready has an established network of committees that provide appropriate communication
channels between academic staff and students. Programme committees meet termly and are
responsible for amongst other things:
‘…monitoring and evaluation of the programme and in particular evaluating its operation,
its delivery and standards, its teaching methods, its curriculum aims and students’
needs’
All TELAs are invited to attend the programme committee and have the opportunity to report
directly to the students regarding the outcomes of the research and act on any feedback the
students may have. In addition to programme committees, the ISS is going to be issued three
times a year creating further scope for a longitudinal analysis of the themes that emerged during
phase one.
Next steps
This is the first year that this research has been carried out and analysed in relation to the
student voice and Moodle. Continuation in subsequent years will provide comparisons between
years and richer analysis. The learning innovation team will be able to cross reference the data
SOLSTICE Conference 2015, Edge Hill University
11
produced with the development of institutional policy, widening the scope of the research
beyond informing faculty based staff development and support.
Undertaking this piece of research has had a substantial impact across the institution. The
outcomes from the thematic analysis have enabled the TELAs and senior members of the
learning innovation team to refresh and revitalise the way in which they support academic
colleagues. Student voice has enabled evidence-based solutions to develop that respond
directly to the distinct needs and aspirations of both students and staff.
Conclusion
The ISS at MMU has provided an extensive student voice data set which provides rich feedback
about the experience of technology enhanced learning, teaching and assessment. By
undertaking a detailed thematic analysis at institutional and faculty levels, the learning
innovation team have been able to target staff development and training provision across this
institution to build on those experiences that students consider to be best practice and work with
academic colleagues to address those issues identified as being in need of
improvement. Overall, in this paper we have shown how this student voice data from the ISS
can be used in the context of a feedback action cycle to determine appropriate responses,
operationalize them and communicate back to the students what has been done.
SOLSTICE Conference 2015, Edge Hill University
12
References
ALDERMAN, L., TOWERS, S. and BANNAH, S. 2012 Student Feedback systems in higher
education: a focused literature review and environmental scan, Quality in Higher Education 18(3)
pp261-280.
HARVEY, L. 2011, ‘The Nexus of feedback and improvement’ in NAIR, C.S. and MERTOVA, P.
(Eds), Student Feedback: The cornerstone to an effective quality assurance system in higher
education. Oxford, Chandos.
LANGAN, A. M., SCOTT, N., PARTINGTON, S. and OCZUJDA, A., 2015 Coherence between
text comments and the quantitative ratings in the UK’s National Student Survey, Journal of Further
and Higher Education, DOI: 10.80/0309877X.2014.1000281
REED, P. and WHATMOUGH, S., 2015 Hygiene factors: Using VLE minimum standards to avoid
student dissatisfaction. E-learning and Digital Media 12(1), pp. 68-89.
RITCHIE, J. and SPENCER, L. (1994) Qualitative data analysis for applied policy research
In Bryman, A and Burgess, R.G Eds Analyzing qualitative data. London: Routledge,
Chapter 9 pp173-194.
STUBBS, M., 2014. Keynote: Enhancing the Student Experience at MMU: Clearing a path and
interpreting digital footprints. European First Year Experience Network (EFYE) 2014
Conference 9 – 11 June,
http://www.ntu.ac.uk/apps/events/9/home.aspx/event/151843/multimedia Nottingham,
Nottingham university
STUBBS, M., 2014. Transforming the Student Experience: Manchester Metropolitan University’s
EQAL project European University Information Systems conference 2014, Umea: Umea
University pp1-6.
SOLSTICE Conference 2015, Edge Hill University
13