Clegg: Effect of in-class electronic response systems on student engagement
Overview:
Electronic response systems, ranging from simple clicker technology to more sophisticated reporting
systems that enable a wider variety of response types, including free responses, have become
increasingly commonplace in large college classrooms, where a traditional dialogue to engage all
students is not feasible. In the course IB 150 we recently switched from clicker technology to a Wi-Fi
enabled student response system that requires students to bring web-enabled devices to class
(smartphone, tablet, or laptop). Previous research has demonstrated the efficacy of response systems
to effectively engage students in large classrooms. At the same time, results have also demonstrated
that presence of web-enabled devices in classrooms can negatively impact student attention and
ultimately performance.
Here we are interested in assessing the impact of web-enabled devices on performance and peer-topeer engagement during in-class group activities. We are proposing to assess the effect of webenabled devices on peer-to-peer interactions using a counter-balanced, split-class design. All
students will have access to paper worksheets. The two treatments are: groups that turn in their
answers on the physical worksheets and groups that report their answers through their web-enabled
devices. Students will switch between treatments for the different activities to ensure that every
student is equally exposed to both treatments. We predict that presence of web-enabled devices will
result in reduced peer-to-peer engagement during activities given physical constraints associated
with entering responses through screen interfaces. Results from this assessment will inform future
decisions on the use of student-response systems in our courses, and will contribute valuable data to
the discussion on the efficacy of such technology in large lecture courses.
Annotated bibliography:
1.
Jensen, J.L., Kummer, T.A., & Godoy, P.D.d.M. (2015) Improvements from a Flipped Classroom
May Simply Be the Fruits of Active Learning. CBE - Life Sciences Education. 14(1) Article 5.
Jensen, Kummer, and Godoy found no significant differences in either performance or student
satisfaction between active classrooms that were not flipped to those that were flipped. The
learning gains achieved by students in both cases is attributed to an active, constructivist course
design.
In our course (IB 150) we have recently transitioned from an active, non-flipped classroom to an
active flipped classroom, and similarly found no change in student performance as measured on
the final exam. The notable difference being additional class time freed for active student
engagement in a flipped classroom design.
This reference and citations therein underline the efficacy of an active, constructivist classroom on
learning outcomes, as has been repeatedly demonstrated. We have recently transitioned to using a
web-enabled student response platform to allow free responses, graphing of predictions, and
numerical responses that enable us to ask higher-order questions while allowing efficient feedback
to students despite very large class sizes (>500 students).
2.
Carnaghan, C., & Webb, A. (2007). Investigating the effects of group response systems on student
satisfaction, learning, and engagement in accounting education. Issues in Accounting Education,
22(3), 391-409.
Carnaghan and Webb found that the use of clickers did not increase class performance on exams,
while students believe their learning was aided by the use of clickers. They explain that the belief
of students in contrast to their performance data might be affected by the overall high satisfaction
of students when engaged in class with clickers. The authors propose a “Halo Effect” of student
satisfaction with the clicker technology that creates a belief of improvement in their learning using
clickers.
This paper highlights improvements in engagement, interaction among students, and student
satisfaction that follow from a more student-focused, active classroom enabled by the use of
clicker questions in class. However, it (and references therein) also raises the question whether the
use of clicker questions actually improves learning outcomes, one of our key goals in our course.
3.
Shieh, R.S. & Chang, W. (2013). Implementing the Interactive Response System in a High School
Physics Context: Intervention and Reflections. Australasian Journal of Educational Technology,
29(5), 748-761.
Shieh and Chang highlight the importance of effective use of high quality questions when using
Interactive Response Systems in courses. The most effective clicker questions are those that
require deeper thinking at higher-order levels of Bloom’s Taxonomy. Questions need to be
challenging enough to allow students to struggle, and to require them to engage with their peers,
and draw on multiple concepts, in the process reinforcing their understanding of these concepts.
Good questions further allow a teacher to identify misconceptions held by students and to target
the specific misunderstandings.
This paper highlights the importance of the nature of student engagement in its effectiveness.
Merely engaging students to answer questions is insufficient to elicit learning gains. This may in
part explain the contrasting results of small performance gains to no significant impact of using
student response systems on learning outcomes among studies such as references in number (1)
above.
4.
Bryfczynski, S.P.; Brown, R.; Hester, J.; Herrmann, A.; Koch, D.L.; Cooper, Melanie M.; Grove,
N.P. (2014). uRespond: iPad as Interactive, Personal Response System. Journal of Chemical
Education. 91(3), 357-363.
Bryfczynski et al. highlight the limitation of clicker-based student response systems to multiplechoice formats, and introduce an example of a web-enabled response system using tablets that
allow students the ability to engage with other question modalities, such as in their case graphing
and visual tasks.
In our class we are currently using a similarly web-enabled platform (although it can be used more
broadly on any web-enabled device) to allow students to engage through free responses, graphing
predictions to their hypotheses, enter numeric values from calculations etc in order to enhance
higher-order engagement opportunities, including application, analysis and synthesis – related
questions.
5.
Sopina, E., & McNeill, R. (2015) Investigating the Relationship between Quality, Format and
Delivery of Feedback for Written Assignments in Higher Education. Assessment & Evaluation in
Higher Education, 40(5), 666-680.
Sopina and McNeill found no effect on student perception of quality, format and timeliness of
feedback received on open-ended essay questions provided electronically versus returned as hard
copy on paper submissions, while noting a strong difference in speed and ease of providing
feedback for instructors.
This paper is of interest to our research question, as it suggests that electronic marking of free
responses can result in more timely feedback for students without impacting on feedback quality
to students. In short: Even if student learning is not impacted, the benefits to course delivery from
the instructor side may make the use of electronic response systems that allow free responses by
students beneficial.
6.
Kay, R.H., & Lauricella, S. (2014) Investigating the Benefits and Challenges of Using Laptop
Computers in Higher Education Classrooms. Canadian Journal of Learning and Technology.
40(2).
Kay and Lauricella found that laptop use in classrooms improved note taking, communicating and
sharing of information with peers, and the interaction with online interactive tools, while also
increasing non-course related distractions, such as surfing webpages, social networking and
watching videos or playing games during the class time.
This paper highlights the motivation for our study. While it is clear that active, student-centered
engagement of students has strong benefits over traditional lecturing, it is less clear to us whether
a web-enabled technology to allow student interaction with challenging activities benefits students
equally to providing the same activities through a paper-based response. While Sopina & McNeil
under reference 4 found no difference in student perception of the two modalities, their study did
not include performance metrics. In contract, Kay and Lauricella raise the question whether paperbased student submissions could be superior in allowing similar levels of engagement while
reducing the likelihood of distraction.
7.
Sana, F., Weston, T., & Cepeda, N.J. (2013) Laptop Multitasking Hinders Classroom Learning for
Both Users and Nearby Peers. Computers & Education. 62, 24-31.
In contrast to the previous studies, Sana, Weston, & Cepeda quantified the effect that in-class
electronic devices have on student performance. The authors found both a decrease of students
using laptops in the course, and also reduced performance of students sitting near other students
with laptops, indicating that the distractive influence has a broad influence in a course.
This study focused on students using laptops for non-class related purposes (except for note
taking). It thus does not directly test the impact of using web-enabled devices in a directed fashion
for classroom activities. Thus, this study provides a wonderful springboard for our research
question, and allows the formulation of two alternative hypotheses:
i.
ii.
Web-enabled electronic submissions provide a comparable learning experience to paperbased submissions for students in an active, flipped classroom (with a focus on problem
solving and application rather than lecturing).
Web-enabled electronic submissions provide a lower-quality learning environment when
compared with paper-based submissions, due to a higher likelihood of their use for
distracting, non-class related purposes.