CAPHIA TEACHING AND LEARNING FORUM
18-19 September 2014, Perth
Evaluating flipping: Development of
the Flipped Classroom Student
Engagement Questionnaire (FCSEQ)
Jane Taylor
Public Health Program
University of the Sunshine Coast
Flipped team
Jane
Taylor
Rachel
Cole
Public Health
Course
Coordinator &
Evaluator
Public Health
Course
Coordinator &
Evaluator
Mary
Kynn
Statistician
& Movie
Maker
Julie-Anne
Foster
Bec
Tretheway
Paramedic
Science
Course
Coordinator
Project
Research
Assistant
Kara Lilly
Public
Health
Course
Coordinator
Exploratory Learning and Teaching Grant, 2013
Centre for Support and Advancement of Learning and Teaching, USC
FCSEQ Movie
18/09/2014
Jane Taylor, University of the Sunshine Coast
Overview
• A bit about flipping
• Research aim and process
• Flipped Classroom Students Engagement
Questionnaire (FCSEQ)
• Some very preliminary (limited) findings
• Comments & questions
18/09/2014
Jane Taylor, University of the Sunshine Coast
What is a flipped classroom?
• Pre-class online learning activities
• Face-to-face workshops
• Post class learning activities
18/09/2014
Jane Taylor, University of the Sunshine Coast
Design
foundations
Bloom’s
taxonomy
SOLO
taxonomy
Introduction
of concepts
Knowledge
Comprehensi
on
Application
Prestructural,
Unistructural,
Multistructural
Clarification
and
exploration of
concepts
Application
Analysis
Synthesis
Evaluation
Multistructural,
Consolidation
of concepts
through
assessment
Application
Analysis
Synthesis
Evaluation
Relational,
http://ctl.utexas.edu/teaching/flipping_a_class/what_is_flipped
18/09/2014
Jane Taylor, University of the Sunshine Coast
Relational
Extended
abstract
Why flip a classroom?
Constructivist pedagogy
•
•
•
•
Student centred
Active and problem based learning strategies
Emphasis on student responsibility for learning
Facilitator role of teacher
Students
•
•
•
•
•
Learn more deeply
Greater participation
Increase in interaction and learning from one another
Greater level of instructor feedback
More time on higher levels of learning e.g., analysis,
synthesis and evaluation
18/09/2014
Jane Taylor, University of the Sunshine Coast
Flipped project overview
2013, S1: Academic team foci
2013, S2: Test run in 3 classes, process feedback
2014, S1&2: Pilot in 10 classes (about 300
students) evaluation
Evaluation: Informal in class feedback , survey
end course, discussion groups end course, SETAC
18/09/2014
Jane Taylor, University of the Sunshine
Coast
Why this research?
• Limited literature on assessing and monitoring
impacts of flipped classroom approach on
student engagement and learning
• Engagement = involvement in purposeful /
meaningful academic activities (Kuh, 2001)
• Aim: To develop and pilot an impact evaluation
questionnaire called the Flipped Classroom
Student Engagement Questionnaire, to evaluate
the impact of the flipped classroom on student
engagement in learning
18/09/2014
Jane Taylor, University of the Sunshine Coast
Flipped Classroom Student
Engagement Questionnaire
Aim
• To assess students’ overall level of engagement at the
course level, and in pre, in class and post class flipped
classroom components
Development
1. Item generation – literature review, existing instruments
and items, peer-review, pilot test
2. Validation (currently here)
3. Finalisation
18/09/2014
Jane Taylor, University of the Sunshine Coast
1. Item generation
• Literature review
– Engagement measures at course level
– Handelsman et al., 2005; Delialioglu, 2012
– National Survey of Student Engagement (NSSE), 2011
• Student engagement constructs & indicators
– Skills, Emotional, Participation and interaction,
Performance (Handelsman et al)
– Active and collaborative learning (Delialioglu, 2012; NSSE)
– Time on task (Delialioglu, 2012)
– Overall level of engagement (Handelsman, et al., 2005)
– Confounding variables
18/09/2014
Jane Taylor, University of the Sunshine Coast
Construct /
Indicators
Students …
Examples
Skills engagement
Study behaviours
Keeping up to date with readings, taking notes in class, and
coming to each scheduled class
Participation/interac
tion engagement
Participation in class and
interact ion with the
lecturer/tutor and fellow
students
Ask questions if they don’t understand the lecturer/tutor;
participate in small group discussions and overall class
discussions and assisted fellow students
Emotional
engagement
Connection with the course
Choosing to make the course material relevant to their lives;
applying the course material to their lives; finding ways to
make the course interesting; thinking about the course
between classes; and really desiring to learn the material.
Performance
engagement
Levels of performance towards
getting a good grade
Doing well in assessment tasks and asking questions,
confidence that they can learn and do well in the course
Global engagement
Levels of engagement between
courses
Flipped compared to traditional
Active and
collaborative
learning
Frequency of particular
learning strategies in and out
of the classroom
Asking questions, completing pre-class activities, assisted
other students, asked for assistance from other students, and
worked with other students on course material
Total time on task
Sum of time in hours doing
specific learning activities
Pre-class activities, watching e-lectures, studying, reading
and writing
Jane Taylor, University of the Sunshine
Coast
Survey examples
Performance
18/09/2014
Jane Taylor, University of the Sunshine Coast
Pre-testing instrument
• Peer-review
– Steering Committee
– Two additional academics
• Pilot test
– OT class using flipped
– Interpretability, usability, timeliness…
18/09/2014
Jane Taylor, University of the Sunshine Coast
1. Item generation
• Confounders (Handelsman et al)
– Entity theory of learning
• Learning can be extended and is not a fixed entity (Handelsman et al)
– Incremental theory of learning
• Certain amount of intelligence and you can’t do much to change it
(Handelsman et al)
– Socio-demographic characteristics
• Age, sex, employment, level of education striving for; level of parent
education; international/domestic student; living arrangements,
access to internet at home, confidence in using technology (NSSE)
• Assessment of flipped approach compared with
usual/traditional
• Perception of impact of flipped approach on academic
performance, non-academic skills, working in groups,
preferences
18/09/2014
Jane Taylor, University of the Sunshine Coast
FCSEQ structure
• Four sections
1.
2.
3.
4.
•
•
•
•
Overall engagement
Participation in course
Achievement
Socio-demographic
19 closed-answer questions
2 open-ended questions
10 minutes
Administration hard copy or electronic
18/09/2014
Jane Taylor, University of the Sunshine Coast
2. Validation
• Administration: four public health courses,
one paramedic course
• Electronic and hard copy administration
• June-August 2014
18/09/2014
Jane Taylor, University of the Sunshine Coast
2. Validation
• Factor analysis may collapse one or more
constructs – in progress
• How do these constructs then align with other
student attributes and measures of
performance?
18/09/2014
Jane Taylor, University of the Sunshine Coast
3. Finalisation & next steps
• Finalise instrument by end October 2014
• Recruitment of more flipped classrooms by end
January 2015
• Full implementation 2015
• Snippets from public health courses Sem 1 2014
18/09/2014
Jane Taylor, University of the Sunshine Coast
Overall engagement in course across
semester?
n=51
18/09/2014
Jane Taylor, University of the Sunshine Coast
Overall flipped or standard classroom?
n=32
18/09/2014
Jane Taylor, University of the Sunshine Coast
Overall level of engagement?
n=51
18/09/2014
Jane Taylor, University of the Sunshine Coast
Overall academic performance?
n=50
18/09/2014
Jane Taylor, University of the Sunshine Coast
Pre class
Doing pre class activities?
(Active & collaborative learning)
n=52
18/09/2014
Jane Taylor, University of the Sunshine Coast
Pre class
Hours per week preparing for workshops
(Total time on task)
n=51
18/09/2014
Jane Taylor, University of the Sunshine Coast
In-class
Workshop attendance?
(Skills engagement)
n=52
18/09/2014
Jane Taylor, University of the Sunshine Coast
In-class
Workshops without pre-class activities?
(Active & collaborative learning)
n=49
18/09/2014
Jane Taylor, University of the Sunshine Coast
In-class
Having fun in workshops?
(Participation / interaction engagement)
n=52
18/09/2014
Jane Taylor, University of the Sunshine Coast
In-class
Active participation on small groups discussion?
(Participation /interaction engagement)
n=52
18/09/2014
Jane Taylor, University of the Sunshine Coast
In-class
Explaining materials to others?
(Active & collaborative learning)
n=50
18/09/2014
Jane Taylor, University of the Sunshine Coast
Post class
Making course material relevant to life?
(Active & collaborative learning)
n=52
18/09/2014
Jane Taylor, University of the Sunshine Coast
Post class
Read additional materials
(Active & collaborative learning)
18/09/2014
Jane Taylor, University of the Sunshine Coast
FCSEQ qualitative feedback
Think more engaged?
 Interaction in class (n=9)
 Having a supportive class learning environment (n=7)
 Pre-class activities assisting in preparing for class (n=4)
Supported engagement?





Getting to know teachers (n=4)
More in-depth learning and expanding on content (n=3)
Ensuring on-track or same page (n=3)
Getting to know/ learning from fellow students (n=3)
Being responsible for/being in control of own learning (n=2)
× Not like pre-class activities (n=2)
× 3 hour workshop too long (n=1)
18/09/2014
Jane Taylor, University of the Sunshine Coast
Comments & Questions
18/09/2014
Jane Taylor, University of the Sunshine Coast
Key References
• Australian Council for Educational Research. Australasian survey of
student engagement: Student Engagement Questionnaire. Australia, 2011.
• Handelsman M, Briggs W, Sullivan N & Towler A. A measure of college
student course engagement. The Journal of Educational Research
2005;98(3): 184-192.
• Delialioglu O. Student Engagement in Blended Learning Environments with
Lecture-Based and Problem-Based Instructional Approaches. Journal of
Educational Technology & Society 2012;15(3): 310.
• Biggs J. & Collis K. Evaluating the quality of learning: The SOLO Taxonomy.
New York: Academic Press, 1982.
• Krathwohl D. A revision of Bloom's taxonomy: An overview. Theory into
Practice. 2002;41(4): 212.
18/09/2014
Jane Taylor, University of the Sunshine Coast