Flipping out about the Flip: All Hype or is there Hope?
Lauren Calimeris1 & Katherine M. Sauer2
Abstract
With increasing pressures to improve student learning outcomes via innovative teaching techniques and with
advances in technology enabling instructors to become more innovative, many instructors are choosing to
“flip” their classroom. This means the activities that traditionally take place in the classroom, such as lecture,
are experienced outside of the classroom, and activities that traditionally take place outside of the classroom,
such as problem solving, take place inside the classroom. While there are anecdotal stories of the
effectiveness of flipping, thus far there are few quantitative studies analyzing the efficacy of flipping on
student outcomes. This study presents experimental evidence of the effectiveness of flipping an introductory
principles of microeconomics course. Model results show that students who experienced the flipped teaching
method scored significantly higher on midterm and final exams than did the comparison group.
Keywords: Economic education, flipped classroom, inverted classroom, teaching, student performance.
JEL codes: A2, A22 (?)
Lauren Calimeris is an assistant professor at St. John Fisher College (email: lcalimeris@sjfc.edu). She thanks Gary
Maggs, David Pate, and Julie Thomas, as well as the Office of Information Technology for technical assistance. Liz
LaChance provided excellent assistance with obtaining student records. She also thanks her students for their
willingness to participate in the study.
2 Katherine Sauer is the Financial Education Program Director at the University of Colorado.
1
1. Introduction
Economics courses are traditionally taught in the lecture format. While many instructors do attempt to
incorporate active learning in their classes, these activities are limited due to the amount of content that must
be covered (Hughes 2012). Additionally, students most often need the instructor’s expertise when trying to
apply knowledge to problems or homework assignments, when the instructor typically is not around
(Bergman and Sams 2012a, Bergman and Sams 2012b). One solution to this problem is the “flipped” class,
also known as the inverted classroom. Flipping or inverting a class means the activities that traditionally take
place in the classroom, such as lecture, are experienced outside of the classroom while activities that usually
take place outside of the classroom now take place inside of it (Bergman and Sams 2012b; Hughes 2012 ).
In a flipped class, instructors typically break down lectures into modular components and create videos up to
20 minutes in length (Bergman and Sams 2012b, Hughes 2012 ). Students watch these videos outside of the
classroom on their own time. Proponents claim it enables students to learn new content at their own pace,
allowing them to watch, pause, or re-watch lecture videos as necessary (Bergman and Sams 2012b; Talbert
2012). Students may be assigned an activity to complete as evidence of watching the videos (e.g. worksheet,
notes outline, online quiz). In class, instructors may use Just-in-Time Teaching techniques to address
confusion students have with material encountered prior to class. The instructor may have students post
online, email in advance, or bring questions with them to class. After a few minutes of discussion, activities
such as problem solving, group work, discussions, simulations, or experiments may take place.
Studies on course flipping identify benefits to student learning and favorable student perceptions about the
technique. Thus far benefits are anecdotal, are not compared to a control group, or do not control for other
factors that may influence any observed gains from flipping the course (Deslauriers et al 2011, Lage et al.
2000, Moravec et al. 2010, Strayer 2012). Furthermore, studies do not address time-on-task differences
between a traditional course and a flipped course.
This study quantifies the educational benefits of flipping the class in an introductory economics course using
a randomized evaluation of this teaching methodology. We randomly assigned two sections of introductory
microeconomics to comparison and test groups. The same instructor taught both groups, and there were no
significant observable differences between academic and demographic characteristics between the two
sections. Student exam scores on two midterms and the final exam are used to determine the effect of
teaching methods on student outcomes.
Ordinary least squares (OLS) results indicate that students in the flipped classroom scored significantly better
on the second midterm and final exam compared to students in the traditional class.
2. Literature Review
[Theoretical foundation of why flipping should benefit student learning. Learning styles – Kolb, Myers Briggs, Reichmann and
Grasha, etc. Active learning, concrete experiences, etc.] To be expanded upon.
Some studies have attempted to investigate the quantitative and qualitative benefits of flipping (though it is
not always called flipping and is referred to as the “inverted classroom” or learning before lecture), and much
of this investigation is in the sciences (Deslauriers et al. 2011, Moravec et al 2010). One study attempted to
investigate the effect of flipping on student learning in a physics classroom (Deslauriers et al. 2011). In this
study, two sections of a large physics class were taught by different instructors, one was a post-doc and one
was a faculty member. The flip consisted of flipping one week of a twelve week course. During the flipped
portion of the class, two different instructors came in to teach the flipped class. At the end of 12 weeks,
students in the flipped class exhibited greater outcomes on the test than students in the traditional class.
After 6-18 months, there was no different in retention of the material. This study did not test if there was a
significant difference in test scores, nor did it control for class demographics or academic abilities (Deslauriers
et al 2011).
One study in biology did attempt to quantify the effect of flipping on student learning and account for
demographic differences when investigating flipping (Moravec et al 2010). The authors compared students
from previous semesters to students who had a portion of their class flipped. For three different class
periods in the semester, students were asked to complete assignments prior to attending class: watching a
narrated PowerPoint video with a note-taking sheet or completing a one-page worksheet based on book
reading. Students were also given the option of taking a quiz on this material prior to coming to the class.
They were incentivized to complete the learning before lecture activities by getting credit to complete these
activities. In class, students engaged in active learning activities on the topics for which they completed the
pre-class assignments. The authors documented no major significant demographic differences between
those students who were in the flipped class versus those who took the class in previous semesters. To assess
the impact of this structure on student learning, the authors compared these students’ results on the final
exam with those of students in previous semesters. The comparison questions were matched by difficulty
level, though they were not the same questions. The authors found that students did complete the out of
class assignments and that they scored 21% higher on questions pertaining to the flipped material versus
previous students (Moravec et al 2010).
In economics, flipping has been investigated at a qualitative level only. Lage et al (2000) investigated the
effect of flipping before it was called flipping. In their study, students in their “inverted” classroom read the
text outside of class and were encouraged to either watch lecture videos that were videos of the actual lecture
that they could watch in the School of Business or make a VHS copy of and watch at home. Alternatively,
students could copy or watch PowerPoint presentations with audio also in the School of business or at home.
Therefore, watching videos or lectures outside of class was completely optional for the students. During the
class, students engaged in experiments or labs, and then worked on worksheets or review questions. Two
instructors taught two different sections in this flipped manner. Instructors reported enjoying teaching the
course in the flipped manner better than in a traditional manner. The authors reported that students tended
to enjoy the flipped structure and the peer group-work part of class. Students did not think the course was
easier, and they reported working hard in it than in their other classes. This study did not investigate the
effect of flipping on learning outcomes, nor did it contain a control group.
Therefore, thus far the research has not included a treatment-control experimental set up. Furthermore, no
study has investigated the effect of a total flip on student learning. This study is the first to do so.
3. Methods
Experimental Design
During the Fall 2013 semester, the instructor taught two 55- minutes sections of introductory
microeconomics back to back: at 9:05 and 10:10 on Mondays, Wednesdays, and Fridays in the same
classroom3. Students enrolling in the two courses did not know of any differences between the courses
except for the time slot. Using a simple coin toss, we randomly assigned the sections to comparison and
treatment groups. The flipped course is considered the treatment group, while the traditional course is
considered the comparison group. On the first day of class, the instructor informed the students that she was
researching the effectiveness of different teaching techniques, but she did not explicitly state what the
different techniques were, nor did she explain which group was considered the treatment group and which
group was considered the control group. All students in both classes indicated willingness to participate in
the study. note
Both groups had access to the same Aplia problems, reading guides for the text, practice tests, and some inclass worksheets. The instructor-provided class notes and lecture material were identical across groups as
well, except that the lectures were pre-recorded for the treatment group to watch prior to class while the
comparison group experienced the lectures during class. Therefore, while the class notes were exactly the
same, including definitions, explanations, and examples, the comparison group had the advantage of being
able to interact with the lecture and ask questions immediately, while the treatment group had to wait until
class to ask questions.
The instructor taught the comparison group as she had traditionally taught introductory microeconomics
courses. She lectured students on the material for a given chapter, perhaps showed them a clip from a movie
or television show that illustrated the concept in the “real world”, and at the end of each chapter, students
completed worksheets in class. Students worked individually, in pairs, or in groups on these worksheets. For
homework, students read the text and completed Aplia assignments.
In the flipped class, students experienced the lectures via videos that the instructor created. The instructor
used Echo 360 to record the lecture videos. The videos were all in a similar format: a PowerPoint shell with a
black background. Definitions, data, and some text were pre-loaded into the PowerPoint presentation, where
items would animate upon a click. Using the screen-capture technology, the instructor recorded a voice-over
of the PowerPoint slide show, and where appropriate, she annotated the slides using a pen-tablet technology.
For example, in the videos, axes and titles of graphs were animated to “draw” on the screen, meanwhile, the
instructor hand drew the supply and demand curves, and shaded in the areas of consumer and producer
surplus by hand, and solved calculations by hand. As another example, table shells were pre-loaded into the
PowerPoint, but the instructor explained how to fill them out and filled them out by hand on the recordings.
To keep the videos “interesting” (as if the content weren’t enough!) the instructor included clip art or other
pictures from the internet when appropriate. Videos ranged in length from 4-21 minutes, with a mean of 11
minutes and a standard deviation of 4 minutes.
Students in the flipped class watched 1-4 videos for homework for the start of the next class. During the first
day of class, the instructor showed the students how to access the videos and how to take notes while
watching. She showed the students how to pause and rewind the videos if need be. She also guided them
through how they might choose to take notes. The instructor incentivized the students to watch and take
notes on the lecture videos by giving them a daily quiz on which they were permitted to use their notes. The
quizzes were based off of the lecture material from the video(s) due that day in class. The quizzes were “5
minute” quizzes that contained five true/false, multiple choice, and/or fill in the blank questions, each
The classroom was a tiered classroom for a maximum of 40 students (?). In fact, the students sat in cushioned chairs
with wheels, and the classroom is better suited for a lecture class as students do not have their own desks which they can
maneuver for activities.
3
question being worth one point. The questions were knowledge-based questions, such that students who had
watched the videos and taken good notes should easily have earned a 4 or 5 out of 5 on the quizzes. Because
the instructor had an attendance policy in the traditional class allowing students to miss 6 classes without
penalty, the instructor dropped the lowest 6 quiz grades. This allowed students to miss classes and not be
penalized, to miss lecture videos, to come unprepared to class if need be, or to simply have a poor quiz score
from time to time and not suffer major consequences.
While the students were taking their daily quizzes, the instructor passed back the previous quizzes and
prepared for that class’s activities. After the quizzes, the instructor asked if there were questions on any of
the material, but usually students did not volunteer questions. After the quizzes, students engaged in different
activities depending on the day and material.
On Mondays and Wednesdays, the class participated in a variety of activities, including experiments, such as
the market experiment for supply and demand, or the tennis ball experiment for production and costs;
analyzing and discussing popular press news articles, such as for positive/normative statements and price
controls; watching video clips from popular television shows (e.g. The Colbert Report) and discussing and
analyzing them; creating mind and concept maps; and completing worksheets relating to each of the activities.
Students also completed worksheets identical to those the traditional class completed for each chapter.
To maintain as much consistency between the two classes, and to best identify the effect of flipping, both
groups had the same Aplia homework. Since the students in the flipped class spent their time outside of class
watching lecture videos, Fridays were designated “Aplia Days” during which students worked individually, in
pairs, or in groups to solve problems. The instructor walked around the room and answered questions or
worked through problems with the students as they requested the help. Sometimes, the instructor worked
with a single student, but more often she worked with pairs of students who were working on the homework
together4.
Data
Since the flipped class had the benefit of working on Aplia during class, and they also had quizzes as a part of
their final grade, the overall course grade was not used to assess effectiveness. Instead, raw scores on
midterm exams and the final exam were used to assess the effectiveness of flipping on student outcomes.
Both classes received the same exams, two mid-semester exams and a final exam. Midterm exams were not
cumulative, but the final exam was. Therefore, in a sense, the final exam indicates what the students learned
in the course as a whole. All exams were structured the same and contained fill-in-the-blank definition
questions, true/false questions, multiple choice questions, and short answer questions.
To avoid unintentional bias in grading, the students were assigned random numbers for each of their exams
and wrote these numbers on the cover sheet for their exams instead of their names. The instructor shuffled
the exams together prior to grading to further ensure that she did not know whose exams she was grading.
Once the grading had been completed the instructor sorted the exams and wrote the students’ names on their
exams.
In Aplia, students were allowed to try each problem up to three different times, and each time they were given a
different version of the problem. Since the versions are not necessarily the same from student to student, students
working together would often have to “solve” and think through two different versions to help one another. While this
made it more difficult for students to work together, it also forced them to think about the problems in a more open
manner and to think through the problems more thoroughly than if they were working on their own.
4
While students in the traditional class were expected to read the text outside of class, many students fail to do
so, and so the time that students in the traditionally taught class spend on the course outside of the class
meeting time might be less than students in the flipped classroom. To assess whether the time difference was
a factor and to control for time use outside of class, at the end of each week both groups reported weekly
time on task estimates using the Qualtrics Survey Tool. (Figure 1) To provide students with confidence that
they would not be penalized or judged for their responses and to encourage honesty in reporting, students
were assigned another random number for anonymity purposes. Since the instructor could not enforce filling
out the survey, she incentivized students to fill out the survey by offering extra credit to those who
consistently did (5 points for all surveys filled out, 4 points for missing 1, 3 points for missing 2 on their final
exam score). The instructor informed the students that she would not look at the survey responses until
after she had submitted semester grades, but prior to doing so, she would look at the tally only to assign the
extra credit points. Therefore, this data may be biased because it is based on recall, suffers from selection
bias, and students may have still felt pressure to overestimate. Therefore, this data is included in sensitivity
analyses and robustness checks only.
Student demographic data were obtained from the university’s records and additional information not
included in school records, such as if the student worked or participated in sports or clubs, type of internet
access, and parental education information was obtained using a Qualtrics survey (Figure 2). Once again, the
instructor offered extra credit to students who filled out this survey. Again, this data is self-reported and may
suffer due to self-selection or faulty recall. These data are therefore included in sensitivity analyses only.
Finally, student opinions regarding the flipped class were obtained using a final Qualtrics surve (Figure 3).
Once again, the instructor offered extra credit to students who filled out this survey. Again, this data is selfreported and may suffer due to self-selection or faulty recall. These data are therefore included in sensitivity
analyses only.
A benefit of using the Echo 360 system is that it tracks the number of students who log in, the videos they
watch, as well as the percent of the videos that they watch. (A weakness that was brought to the instructor’s
attention after the conclusion of the semester was that is it currently unable to track students who log in via
an iPhone or iPad). As a robustness checks, we can control for the number of videos the student watched,
the percent of videos that were completed, and the number of times students watched or rewatched videos.
An additional benefit of using Aplia is that Aplia tracks the amount of time that students are logged into
Aplia. Therefore, a comparison of self-reported time on Aplia (Reading the text or answering the problem
sets) and the “official” time on Aplia that is tracked through the program can provide a rough estimate as to
how accurate students recollection of their time spent on various activities is. (Note: a problem with this
“logged in” aspect is that students may log into Aplia, open a new window or tab, and check email or other
social networks while still logged in.) Therefore, this comparison is a guide only.
3. Model
To assess the effect of flipping the class on student outcomes, the following model was estimated:
𝐸𝑋𝐴𝑀𝑖 = 𝛼 + 𝛽𝐹𝐿𝐼𝑃𝑖 + 𝑆𝑖′ 𝛾 + 𝑋𝑖′ 𝛿 + 𝑢𝑖
Where EXAM is the score on student i’s first midterm, second midterm, or final exam, for students i=1…n.
FLIP is a dummy variable equal to one if the student was enrolled in the flipped class and equal to zero if the
student was enrolled in the traditional class. S is a vector of academic controls, including if the student
received AP or college credit in high school, if the student took AP economics, and the student’s math and
reading SAT scores. All of these controls are expected to be positively related to exam scores, as more able
students will generally take AP courses and score higher on SATs compared to students with lower abilities.
X is a vector of demographic controls, including gender, age, ethnicity, highest level of maternal and paternal
education, and household income. The effects of gender and age are ambiguous. On the one hand, females
generally do better in college than males (REF), but economics is also a male-dominated field. Typically, a
priori expectations would suggest age to have a positive and significant effect on test scores, as older students
tend to either take school more seriously, work harder, or generally know how to study effectively compared
to younger students.
If there is a positive effect of flipping on test scores, one might argue that flipping is merely causing students
to spend more time on economics outside of class. Furthermore, one might also argue that if students have
the ability to watch and rewatch the lectures, it might not be the “flipping” that is increasing the test scores,
but rather the ability to watch lectures as many times as desired. Therefore, as robustness checks, we estimate
the following model:
𝐸𝑋𝐴𝑀𝑖 = 𝛼 + 𝛽𝐹𝐿𝐼𝑃𝑖 + 𝑇𝑂𝑇𝑖′ 𝜏 + 𝑉𝐼𝐷𝑖′ 𝜑 + 𝑆𝑖′ 𝛾 + 𝑋𝑖′ 𝛿 + 𝑢𝑖
Where TOT is a vector of time-on-task controls, including the self-reported time spent watching videos,
reading the book, studying for exams, completing homework, and other activities. VID is a vector of
controls for the frequency of lecture video watching. This includes information on the total number of
lecture videos watched, the number of repeated video watches, and the percentage of videos completed.
In addition to assessing the quantitative impact of flipping on student outcomes, student perceptions of class
and learning are important as well. Therefore, to assess the student reception and perceptions of flipping, we
asked students to fill out a survey regarding their opinions and experiences.
4. Results
Table 1 presents the summary statistics for the two classes. There were 78 students enrolled in the two
classes, 38 in the flipped class and 40 in the traditional class. Students in the flipped and traditional classes
were similar in ability, with no significant differences between the groups between taking AP economics and
receiving credit, combined SAT score, and math and reading individual SAT scores5.
There are virtually no significant differences with respect to the demographics between the two classes.
There were no differences with respect to gender, ethnicity, parental levels of education, and parental income
level. The only significant difference between the two groups is that the students in the flipped class were
roughly 6 months older than those in the traditional class (significant at the 10% level). Therefore, it appears
that the two groups are very similar based on observable characteristics and that there was no systematic
sorting between the two classes.
Not all students have SAT or high school GPA data. These students may have transferred in, in which cases this
information was not available.
5
Comparing the self-reported student demographic data, it appears that students in the flipped class were
significantly less likely to live on campus, and, as a result, more likely to have broadband internet at their
residence. Students in the flipped class were also more likely to have fathers whose highest level of education
attained was a high school diploma versus in the traditional class, and they were less likely to have fathers
with a four year college degree compared to those in the traditional class6.
Finally, there are no significant differences between the self-reported weekly time on task reports between the
flipped and traditional classes. Students in the traditional class report spending an average 4.4 hours per week
working on economics outside of class, while students in the traditional class report spending 4.6 hours per
week working on economics outside of class. (p=0.761).
4.1 Empirical Results
The main OLS results are presented in Table 2. The Model I reports the results of flipping on each exam,
controlling for student ability measures. Model II controls for other student demographics, while Model III
incorporates family income controls. Finally, Model IV controls for parental education levels. Generally,
Controls in all specifications mostly exhibit the expected signs. Students with AP credit generally score
higher on economics exams, though this effect is not consistently significant. In the model with the full ste
of controls, it is significant for both Exam 1 and the Final exam. Students who scored higher on the Math
portion of the SAT also have higher scores, with a 10 point increase in the MATH SAT increasing test scores
by roughly 10 percentage points. This conforms to a priori expectations as economics is highly math-based.
Interestingly, the effect, while insignificant, of having taken AP economics is negative across all exams and
specifications. This suggests that either students were complacent and thought they already knew the
material, or topics covered in AP economics may have differed from those in principles of microeconomics.
Interestingly, there was no significant effect of being a Freshman on the first two exam scores, though
Freshman becomes significant for the final exam in the first three models (not controlling for parental
education) at the 10% level. It could be that either freshmen are adjusting throughout the semester, and have
finally “figured it out” by the final exam, or they are motivated to study harder for the final to achieve their
optimal grade given the weight of the final exam. Females generally scored higher than males, though this
effect was significant only at the 10% level for the first exam. Family and household controls are generally
insignificant, and an F-tests for joint significance of income and parental education yields and F-statstic of
1.66 (p=0.1762).
In nearly all specifications, flipping does not have an effect on the first midterm exam (except for in model II,
where it is significant at the 10% level, though the magnitude of the estimate is similar to other models). This
suggests that either students did not take flipping seriously, in that they did not watch the videos, did not pay
attention to the videos even if they were watching them, or did not take good notes on the videos. After the
first exam, students may have realized that they do have to do the work and carefully watch and take notes on
the videos. We address this in a robustness check.
In Model I, controlling only for academic ability, flipping is associated with a 7 point gain on the second
midterm and nearly an 8 point gain on the final exam. The magnitude of the estimates are similar across
Model II. When controlling for family income, the point estimates increase slightly. In the full specification,
Self-reported data was collected via a survey that the students opted to take for extra credit points on their final exam
grade.
6
the results indicate no significant effect of flipping on the first midterm, but a 9 point increase on the second
midterm, and a 9.8 point increase on the final exam. The flipping effect could be due to re-watching videos
as the semester progresses or because the active learning activities in class helped students to remember
concepts more easily. (Thirty-one percent of students responding to opinion survey reported having reviewed
the video material prior to exams). Furthermore, students in the flipped class were able to work on problems
together, and for each activity, they solved additional problems and questions that the traditional class did
not. Therefore, overall, they had more experience solving problems than the traditional class. We
hypothesize that flipping effect might increase as the semester progresses because students actually watch the
videos, and as the final exam approaches, they have the ability to rewatch videos from the lectures.
Table 3 incorporates the student survey responses to demographic questions. Keeping in mind that the
students voluntarily responded to the survey, the results change a little. For example, having taken AP
courses in high school is no longer significant, nor is Math SAT scores. Critical Reading SAT scores are
significant for some examinations, though mostly at the 10% level. Interestingly, this model suggests that
Freshmen score significantly higher on exams than upper classmen. In Model II, controlling for student
demographic and academic factors (e.g., not household or family level characteristics), Freshmen scored 19
points higher on the second exam and nearly 28 points higher on the final exam. In the full specification,
Model IV, Freshmen scored nearly 27 percentage points higher on the final exam.
Students who lived on campus scored higher on both the second midterm and final exam. This could be
because it is easier for these students to form study groups among their friends in the dorms and they have
easier access to study spaces on campus. Students who work scored significantly lower on the second
midterm and final exam. In the full specification, they scored roughly 12 points lower on the second midterm
and 17 points lower on the final exam compared to their non-working counterparts. Working more hours
was also associated with higher test scores. Finally, in the full specification, parental education appears to
influence exam scores on the final exam, though the effects for mother’s education nearly offsets that of
father’s education.
The effect of flipping is even more pronounced in this specification. Being in the flipped class is associated
with gains in student test scores of roughly 9.5 points on exam 1, 14 points on exam 2, and 15 points on the
final exam. These effects are not only statistically significant, but also economically significant as well. A
fifteen point difference is a letter grade and a half on an exam. Identifying the mechanism through which
flipping affects student outcomes could help improve student learning even more.
4.2 Student Perceptions
In addition to assessing the quantitative benefits of flipping, this paper also seeks to assess the qualitative
benefits that may accrue to students. As such, we administered (another) survey to students asking questions
regarding their perceptions of the flipped class (Figure 3).
Seventy-six percent of students enrolled in the flipped class responded to the survey. Of these, forty-five
respondents were female. The majority of respondents were Freshmen (66%), while 28% were sophomores,
and 7% were Juniors. While nearly half (48%) reported that they thought they spent the same amount of
time working in the flipped versus traditional class, 45% reported that they thought they spent more time
working in the flipped format.
To understand how students perceived the course and to determine what might need to change in future
flipped courses. Of the survey respondents, 76% either agreed or strongly agreed that the lecture videos were
helpful in understanding course content, and 79% agreed or strongly agreed that they found it helpful to be
able to watch the videos at a time that was convenient to them. Sixty-nine percent agreed or strongly agreed
that they found it helpful to re-watch portions of the recorded video multiple times until they understood.
Given the importance of the videos, in that this was the main method through which students were first
exposed to the material, these positive responses indicate the videos were useful and utilized in helping
understand course concepts.
Part of the theory behind the flipped classroom suggests that students may learn from one another as they
work together on problems or projects. Roughly 48% of students report agreeing or strongly agreeing that
they did benefit from interacting with peers during class activities (compared to nearly 21% who disagreed or
strongly disagreed). Meanwhile, roughly 38% of students agreed or strongly agreed that they learned new
ways to solve problems by working with their peers in class, compared to 36% who disagreed or strongly
disagreed. Therefore, while a majority did find the videos useful, fewer students saw the benefit of the in
class activities. This could be because they are unware of the benefits, or perhaps they did not enjoy working
in groups and with peers, or the activities may not have been the best fit for some concepts.
Even though students report working harder and longer in the flipped class format, nearly 52% of students
agreed or strongly agreed that they would sign up for this class format in the future, compared to 34.5% who
disagreed or strongly disagreed. Finally, nearly 52% agreed or strongly agreed that they preferred this class
format compared to a traditional lecture class, compared to 31% who disagreed or strongly disagreed.
Student open-ended comments were generally positive. Numerous students liked that they could learn at
their own pace and that they could rewatch the videos if something was unclear or confusing to them.
Students
Therefore, it appears that a majority of students enjoyed the class format and would sign up for a similar class
in the future. Students commented that the videos were lengthy, and they would like, shorter videos that
were more to the point. Students learned about themselves as learners in the flipped class. Some reported
that they learned that they can take efficient notes, some learned that they can be responsible learners, and
some learned that they
5. Discussion
This paper found positive and significant effects of flipping on student outcomes and has documented
positive responses on behalf of the students. As instructors, we enjoyed flipping our classroom. The upfront
costs were high, but day-to-day work was not outrageous, and being more engaged with the students enabled
us to get to know the students better. We were able to target discussions to students’ needs. In all, the flip
examined in this paper was not “perfect”. There were a number of activities that took longer than
anticipated, that did not “properly” illustrate the economic concept that was intended, or that were “boring”.
Therefore, we argue that the estimates in this paper could be considered a lower bound on the effectiveness
of flipping, given that the flip was successful given the inherent trial and error of restructuring and
redesigning a course.
Furthermore, since the videos contained the exact same notes as the live lectures, videos were lengthy. Given
that the optimal length of a video is reported to be roughly 5 minutes (REF), including shorter videos would
be beneficial. The most common student complaint in the flipped structure was that the videos were too
long. Furthermore, encouraging more reading and not feeling the need to explain everything would also
enable the instructor to shorten the videos.
Finally, this study analyzes the effect of flipping only. It is unclear if the videos are the main driver, or if the
active learning activities in class are driving the results, if more time is spent working on economics in the
flipped format, or if it is a combination of all of these aspects. In future analyses, we will control for the
Echo viewing and time on task.
6. Conclusions
Flipping the classroom is a current buzzphrase in academic circles. Many instructors are feeling pressure
from their institutions to implement “innovative” teaching, one such idea being that of flipping. While there
are many anecdotes about how instructors have found flipping improved outcomes or classroom culture, to
our knowledge, no study has implemented a controlled experiment to quantitatively gauge the benefits of
flipping. This study is the first to do so.
This study implemented a treatment-control experimental design to assess the quantitative effects of flipping
a course. Students in the study were unaware of the course structure in the section in which they enrolled
when they enrolled in the course. Courses were taught back to back, and there were no major significant
differences between the two courses. Students in both classes received the same lecture notes, except the
flipped students experienced the content via videos while the traditional class received the content live. They
also received some of the same worksheets, the same online homework, and the same exams. Students in the
flipped class engaged in active learning activites, including online homework and other activities, individually,
in pairs, or in groups, during class and watched lecture videos outside of class. Students in the traditional
class experienced traditional lecture during class, and completed online homework outside of class. Flipping
the class was associated with roughly 10-15 point gains on test scores compared to students in the
traditionally taught class.
Inverting the classroom was associated with significant gains in student performance on exams. While some
students reported truly enjoying the class structure, others claim that they did not enjoy the class structure
and prefer a traditional class. If these same students are the ones who did not benefit from the flipped
environment, colleges in the future could identify flipped courses in descriptions, course titles, or course
numbers, allowing students to self-select into the most appropriate class format for them, improving overall
student outcomes across the board. Furthermore, a successful flip requires student participation, and
students who actively choose to enroll in such a course could exert positive externalities on other students in
the course, increasing the effect of flipping overall.
6. References
Bergman, Jonathan, and Aaron Sams. (2012a). Before you flip, consider this. Phi Delta Kappan 94(2):25.
Bergman, Jonathan, and Aaron Sams. (2012b). Flip your classroom: Reach every student in every class every
day. Eugene, Or. International Society for Technology in Education.
Deslauriers, Louis, Ellen Schelew, and Carl Wieman. (2011). Improved learning in a large-enrollment physics
class. Science 332:862-864.
Hughes, Heather. (2012). Introduction to Flipping the College Classroom. In T. Amiel & B. Wilson (Eds.),
Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2012 (pp. 2434-2438).
Chesapeake, VA: AACE.
Lage, Maureen J., Glenn J. Platt, and Michael Treglia. (2000). Inverting the classroom: A gateway to creating
an inclusive learning environment. Journal of Economic Education 31(1):30-43.
Moravec, Martin, Adrienne Williams, Nancy Aguilar-Roca, and Diane K. O’Dowd. (2010). Learn before
Lecture: A Strategy That Improves Learning Outcomes in a Large Introductory Biology Class. CBE Life
Sciences Edition. 9:473-481.
Strayer, Jeremy F. (2012). How learning in an inverted classroom influences cooperation, innovation, and
task orientation. Learning Environments Research. 15:171-193.
Talbert, Robert. (2012). Inverted classroom. Colleagues 9(1):18-19.
Table 1. Summary statistics and differences between flipped and traditional class.
Flipped Class
Traditional Class
Mean Std Err n
Mean Std Err n
Senior (=1 if Senior)
0.00
0.03
38
0.03
0.02
40
Junior ( =1 if Junior)
0.03
0.03
38
0.00
0.02
40
Sophomore (=1 if Sophomore)
0.21
0.08
38
0.10
0.06
40
Freshman (=1 if Freshman)
0.76
0.09
38
0.88
0.06
40
Enrolled with AP Credit
0.39
0.11
38
0.58
0.08
40
Took AP Econ in High School
0.13
0.08
38
0.13
0.05
40
Math & Reading Combined SAT
1065
27
31
1043
18
36
Math SAT
547
16
31
544
11
36
Reading SAT
517
16
31
499
11
36
Work (=1)*
0.49
0.12
35
0.40
0.08
35
Hours Worked*
6.74
2.16
35
5.97
1.53
35
Club Member (=1)*
0.57
0.12
35
0.53
0.09
34
Live On Campus (=1)*
0.43
0.11
35
0.77
0.08
35
Broadband Internet (=1)*
0.49
0.11
35
0.17
0.08
35
Satelite Internet (=1)*
0.03
0.03
35
0.00
0.02
35
Other Internet (=1)*
0.06
0.06
35
0.06
0.04
35
Female (=1)
0.26
0.11
38
0.43
0.08
40
Age (years)
18.84 0.32
38
18.30 0.22
40
White (=1)
0.87
0.09
38
0.75
0.06
40
Hispanic (=1)
0.03
0.04
38
0.05
0.03
40
Black (=1)
0.00
0.04
38
0.05
0.03
40
Asian (=1)
0.05
0.04
38
0.00
0.02
40
American Indian (=1)
0.00
0.03
38
0.03
0.02
40
Mother's Education: College (=1)
0.58
0.11
38
0.45
0.08
40
Mother's Education: High School (=1)
0.29
0.11
38
0.43
0.08
40
Mother's Highest Education: Unknown (=1)
0.13
0.08
38
0.13
0.05
40
SR Mother's Highest Ed: Less HS(=1)*
0.00
0.03
35
0.03
0.02
35
SR Mother's Highest Ed: : High School (=1)*
0.29
0.10
35
0.14
0.07
35
SR Mother's Highest Ed: Some College (=1)*
0.14
0.10
35
0.29
0.07
35
SR Mother's Highest Ed: 2 College Degree (=1)*
0.09
0.06
35
0.06
0.04
35
SR Mother's Highest Ed: 4 Yr College Degree (=1)*
0.37
0.11
35
0.31
0.08
35
SR Mother's Highest Ed: Graduate Degree (=1)*
0.11
0.08
35
0.17
0.06
35
Father's Education: College (=1)
0.32
0.11
38
0.40
0.08
40
Father's Education: High School (=1)
0.55
0.11
38
0.45
0.08
40
Father's Highest Education: Unknown (=1)
0.13
0.08
38
0.15
0.06
40
SR Father's Highest Ed: Less HS(=1)*
0.00
0.04
35
0.06
0.03
35
SR Father's Highest Ed: High School (=1)*
0.46
0.10
35
0.14
0.07
35
SR Father's Highest Ed: Some College (=1)*
0.14
0.09
35
0.23
0.07
35
SR Father's Highest Ed: 2 College Degree (=1)*
0.23
0.11
35
0.34
0.08
35
SR Father's Highest Ed: 4 Yr College Degree (=1)*
0.03
0.07
35
0.14
0.05
35
SR Father's Highest Ed: Graduate Degree (=1)*
0.14
0.08
35
0.09
0.05
35
First Generation Scholar (=1)
0.29
0.11
38
0.38
0.08
40
High Income Family (=1)
0.47
0.11
38
0.43
0.08
40
Middle Income Family (=1)
0.16
0.10
38
0.30
0.07
40
Low Income Family (=1)
0.24
0.09
38
0.20
0.07
40
*Student-reported
Difference
Mean Std Err t-stat N total
-0.03 0.03 -0.97
78
0.03
0.03
1.03
78
0.11
0.08
1.35
78
-0.11 0.09 -1.28
78
-0.18 0.11 -1.60
78
0.01
0.08
0.09
78
21.51 26.76 0.80
67
3.25 15.71 0.21
67
18.25 15.75 1.16
67
0.09
0.12
0.71
70
0.77
2.16
0.36
70
0.04
0.12
0.35
69
-0.34 0.11 -3.08
70
0.31
0.11
2.93
70
0.03
0.03
1.00
70
0.00
0.06
0.00
70
-0.16 0.11 -1.50
78
0.54
0.32
1.71
78
0.12
0.09
1.32
78
-0.02 0.04 -0.54
78
-0.05 0.04 -1.40
78
0.05
0.04
1.47
78
-0.03 0.03 -0.97
78
0.13
0.11
1.13
78
-0.14 0.11 -1.24
78
0.01
0.08
0.09
78
-0.03 0.03 -1.00
70
0.14
0.10
1.46
70
-0.14 0.10 -1.46
70
0.03
0.06
0.46
70
0.06
0.11
0.50
70
-0.06 0.08 -0.68
70
-0.08 0.11 -0.77
78
0.10
0.11
0.90
78
-0.02 0.08 -0.23
78
-0.06 0.04 -1.44
70
0.31
0.10
3.01
70
-0.09 0.09 -0.91
70
-0.11 0.11 -1.05
70
-0.11 0.07 -1.72
70
0.06
0.08
0.74
70
-0.09 0.11 -0.79
78
0.05
0.11
0.43
78
-0.14 0.10 -1.49
78
0.04
0.09
0.39
78
Table 2. OLS analysis of flipping on Test Scores
Model I
Model II
Model III
Model IV
exam1raw exam2raw finalraw exam1raw exam2raw finalraw exam1raw exam2raw finalraw exam1raw exam2raw finalraw
Flipped
AP Econ (=1)
AP or College Credit (=1)
Math SAT
Crit Read SAT
Freshman (=1)
Female (=1)
Age (years)
White (=1)
Family Income: High (=1)
Family Income: Mid Level (=1)
3.3156
7.0674** 7.5628** 5.6053* 7.7230** 8.2175**
5.6258
8.0394**
(3.434)
(3.126)
(3.037)
(3.341)
(3.121)
(3.162)
(3.461)
(3.170)
-5.9682
-0.8643
-5.0337
-6.0836
-0.8427
-4.8673
-6.3429
-1.4022
(5.043)
(4.149)
(4.459)
(4.688)
(4.214)
(4.459)
(4.629)
(3.946)
6.8840*
5.6989* 8.9317*** 7.3105*
5.4339
9.0269** 7.3459*
5.6111
(3.836)
(3.400)
(3.248)
(3.697)
(3.546)
(3.418)
(3.757)
(3.483)
0.0853*** 0.0958*** 0.0876*** 0.1045*** 0.1014*** 0.0911*** 0.1095*** 0.1044***
(0.032)
(0.027)
(0.031)
(0.031)
(0.028)
(0.032)
(0.033)
(0.029)
0.0392
0.0160
0.0127
0.0430
0.0179
0.0072
0.0458
0.0299
(0.029)
(0.023)
(0.026)
(0.031)
(0.023)
(0.027)
(0.034)
(0.023)
-2.0474
-1.3446
7.4933*
9.8744
0.9065
14.8368* 10.2979
3.3578
(4.389)
(5.780)
(4.197)
(7.114)
(7.566)
(7.820)
(8.083)
(7.815)
7.1712*
2.8404
2.7579
7.1155*
2.7630
(3.811)
(3.211)
(3.052)
(3.857)
(3.097)
3.4621
0.2939
2.8404
3.4294
0.4097
(3.047)
(3.058)
(2.567)
(3.231)
(3.073)
-7.7239
-1.7244
1.0475
-7.9319
-2.0975
(4.964)
(3.357)
(4.376)
(4.898)
(3.237)
-3.1642
-5.7334
(4.381)
(3.694)
-1.4870
-0.0619
(5.446)
(3.847)
Mother's Highest Ed: College (=1)
Father's Highest Ed: College (=1)
Constant
-4.8796
(17.332)
Observations
66
R-squared
0.279
Robust standard errors in parentheses
*** p<0.01, ** p<0.05, * p<0.1
2.4455
(12.570)
-4.2028
(14.274)
-88.9602
(66.982)
-8.8043
(61.242)
-63.9178
(56.798)
-90.8633
(70.953)
-17.9832
(61.371)
66
0.351
66
0.362
66
0.364
66
0.363
66
0.377
66
0.371
66
0.400
8.3264**
5.2787 9.0530***
(3.158)
(3.535)
(2.879)
-4.7727
-6.2349
-1.0241
(4.512)
(4.723)
(3.814)
9.0571** 7.2393*
5.2564
(3.426)
(3.850)
(3.489)
0.0859** 0.1076*** 0.1077***
(0.036)
(0.033)
(0.029)
0.0087
0.0396
0.0234
(0.029)
(0.036)
(0.027)
15.3398* 10.2385
-0.4737
(8.131)
(8.975)
(9.105)
2.7967
7.3326*
2.7213
(3.063)
(3.871)
(3.144)
2.9320
3.2859
-0.7487
(2.573)
(3.409)
(3.372)
1.1558
-7.8128
-2.3782
(4.606)
(4.946)
(3.161)
1.6243
-4.8844 -9.6850*
(3.922)
(5.119)
(5.425)
1.8896
-2.8187
-1.4153
(4.704)
(5.815)
(4.111)
2.8332
-2.3424
(3.628)
(4.195)
0.7514
7.6240
(4.357)
(5.851)
-65.3924 -84.5398
8.5052
(57.161) (75.768) (69.261)
66
0.380
66
0.378
66
0.433
9.7765***
(3.379)
-4.3537
(4.254)
8.6669**
(3.365)
0.0910**
(0.035)
0.0037
(0.031)
10.5634
(8.664)
2.6328
(2.987)
1.5525
(2.809)
0.7425
(4.542)
-2.4493
(5.586)
0.8761
(4.518)
-4.3949
(3.798)
9.1788*
(4.694)
-35.4101
(63.679)
66
0.427
Table 3. OLS analysis of flipping on Test Scores with Student Reporting
Model II
Model III
exam1raw
exam2raw
finalraw
exam1raw
exam2raw
9.4296**
(4.305)
-5.9422
(5.891)
3.9925
(4.330)
0.0716
(0.047)
0.0701*
(0.035)
19.8561
(12.359)
12.6309***
(2.950)
-0.5236
(4.001)
4.3081
(2.917)
0.0495
(0.032)
0.0603**
(0.027)
19.0951**
(8.934)
11.8721***
(3.896)
-2.4208
(4.300)
6.6958*
(3.877)
0.0536
(0.041)
0.0415
(0.031)
27.8796**
(11.670)
9.3820**
(4.553)
-6.0640
(5.906)
4.1049
(4.495)
0.0741
(0.055)
0.0705*
(0.038)
19.8396
(14.084)
13.1467***
(3.041)
-1.0504
(3.914)
4.4570
(2.971)
0.0387
(0.034)
0.0724**
(0.028)
21.6040**
(9.340)
Live On Campus (Y=1)
3.4301
(5.057)
6.6614*
(3.566)
8.0462*
(4.020)
3.3240
(5.383)
7.2161*
(3.744)
Works (=1)
-9.1989
(9.308)
1.1926*
(0.693)
4.8470
(4.948)
2.7918
(4.939)
7.9953*
(4.042)
0.9206
(4.973)
-6.6460
-10.6300**
(4.784)
1.4480***
(0.384)
4.8903
(3.497)
4.2524
(3.049)
1.2917
(3.524)
0.0744
(4.043)
-3.2018
-12.1431*
(6.671)
1.3894**
(0.582)
2.7128
(4.535)
3.9815
(4.815)
0.6923
(4.150)
1.7593
-3.855
0.8911
-8.8353
(9.742)
1.1678
(0.736)
4.7697
(5.202)
2.6035
(5.538)
8.0438*
(4.293)
0.8922
(5.297)
-6.6582
(6.042)
(3.874)
(5.212)
(6.287)
Flipped
AP Econ (=1)
AP or College Credit (=1)
Math SAT
Crit Read SAT
Freshman (=1)
Hours worked
Athelete (=1)
Clubs (=1)
Female (=1)
Age (years)
White (=1)
Model IV
finalraw
12.5338**
*
(3.791)
-1.9077
(4.288)
6.0069
(3.826)
0.0294
(0.044)
0.0466
(0.031)
29.5961**
(11.649)
exam1raw
exam2raw
9.5212**
(4.571)
-6.0665
(5.693)
3.9096
(4.482)
0.0672
(0.055)
0.0603
(0.040)
20.2859
(13.470)
14.1537***
(3.000)
-0.5998
(3.765)
3.8228
(2.929)
0.0350
(0.035)
0.0704**
(0.030)
20.7405**
(9.730)
3.2064
(5.556)
7.7313**
(3.706)
-11.1888**
(5.192)
1.4798***
(0.400)
4.9248
(3.480)
4.6287
(3.267)
1.1904
(3.498)
0.4142
(4.240)
-3.6224
9.1404**
(3.869)
15.0354**
(6.923)
1.5827**
(0.595)
3.2741
(4.513)
5.5347
(5.153)
0.2892
(4.167)
2.1739
(3.746)
0.7078
finalraw
14.9196**
*
(3.973)
-0.7681
(4.115)
4.6237
(3.541)
0.0275
(0.041)
0.0528*
(0.030)
26.9221**
(11.186)
10.5707**
*
(3.793)
-8.6911
(9.485)
1.2124
(0.724)
4.3412
(5.156)
3.0173
(5.435)
8.3264*
(4.448)
0.7657
(5.674)
-6.9583
-12.0197**
(4.879)
1.6250***
(0.388)
4.5272
(3.471)
5.2862*
(3.079)
1.0954
(3.423)
-0.4131
(4.376)
-3.9172
-17.2917**
(6.649)
1.8993***
(0.610)
2.7470
(4.044)
6.7339
(4.713)
-0.2641
(3.930)
0.2274
(3.254)
0.2973
(4.165)
(5.451)
(6.319)
(4.108)
(4.942)
Table 3. OLS analysis of flipping on Test Scores with Student Reporting (cont)
Model II
exam1ra exam2ra
finalra
exam1ra
w
w
w
w
Family Income: High (=1)
-0.9973
(5.277)
-0.4527
(7.069)
Family Income: Mid Level (=1)
Model III
exam2ra
w
finalraw
exam1ra
w
Model IV
exam2ra
w
-1.1041
(3.865)
2.9893
(4.147)
6.2505
(4.208)
5.0646
(4.905)
-4.6302
(6.024)
-2.4740
(7.608)
-4.1219
(3.919)
2.1880
(3.720)
2.3472
(4.140)
-1.9303
(3.193)
4.1666
(4.894)
6.0435*
(3.444)
2.6704
(5.111)
5.2871
(4.502)
7.4754**
(3.508)
10.6205*
*
(4.687)
Mother's Highest Ed: College
(=1)
Father's Highest Ed: College
(=1)
Constant
Observations
R-squared
Robust standard errors in parentheses
*** p<0.01, ** p<0.05, * p<0.1
-56.4252
-25.1952
(99.124)
(74.537)
63.017
5
(78.657
)
59
0.406
59
0.557
59
0.491
finalraw
-56.7143
-34.6252
-67.0190
-46.6802
-16.4328
-32.2637
(105.206)
(76.825)
(76.029)
(111.372)
(76.443)
(66.407)
59
0.406
59
0.570
59
0.519
59
0.422
59
0.591
59
0.585
Table 4. Student Perceptions of Flipped Class Format: Percent of Students Responding
Question
I found the video lectures helpful in understanding course content.
It was helpful to be able to watch the video lectures at a time that
was convenient for me outside class.
I found it helpful to rewatch portions of the recorded video lectures
multiple times until I understood the material.
I benefited from the classroom activities that allowed me to interact
with my peers to solve problems.
I learned new ways to solve problems by seeing how my classmates
solved problems during class activities.
I found it easier to take notes in this class than in a traditional class.
I prefer this class format compared to a traditional lecture class.
I would sign up for another class taught in this format in the future.
Strongly
Agree
27.6%
Agree
48.3%
Neither
Agree nor
Disagree
13.8%
41.4%
37.9%
13.8%
6.9%
0.0%
31.0%
37.9%
27.6%
3.4%
0.0%
20.7%
27.6%
31.0%
13.8%
6.9%
17.2%
20.7%
27.6%
20.7%
13.8%
20.7%
31.0%
31.0%
24.1%
20.7%
20.7%
24.1%
17.2%
13.8%
27.6%
17.2%
24.1%
3.4%
13.8%
10.3%
Strongly
Disagree Disagree
3.4%
6.9%
Figure 1. Time on Task Survey*
Time-on-Task Survey
Econ ID # ______________
Week of ___________________
Please fill out to the best of your memory, as truthfully and honestly as possible.
Last week, how much time did you spend on each of the
following course activities, outside of our class time together?
Reading the book (including taking notes while you read)
Re-copying or studying lecture notes
Working on required Aplia problems
Working on practice problems
Reviewing/resolving in-class assignments
Making flash cards
Solving practice tests
Other Activities(please list)
______ hours and ______ minutes
______ hours and ______ minutes
______ hours and ______ minutes
______ hours and ______ minutes
______ hours and ______ minutes
______ hours and ______ minutes
______ hours and ______ minutes
______ hours and ______ minutes
Week’s Total ______ hours and ______ minutes
Please comment on the time you put into this course last week, outside of our class meetings. Are
you satisfied with your strategy? Do you wish you could change something? Additionally, if there
was a reason that prevented you from studying as much as you would have liked, please explain (e.g.
suddenly getting called into work, just couldn’t find the motivation, had a different class’s test to
study for, etc).
*Note: Actual survey was converted to online format, but contained the same questions. Survey is available
in online format upon request.
Figure 2. Student Demographic Survey
Please answer the following questions to the best of your knowledge. Thank you!
Do you work during the school year?
Yes No
If you work during the school year, on average, how many hours per week do you work?
Hours
Do you play a sport at Fisher?
Yes No
Do you belong to any clubs at Fisher?
Yes No
What type of internet access do you currently have at your off-campus residence during the semester?
N/A, I live on campus
None
Dial up
High Speed/Broadband
Satellite
Other
What is your age in years?
17
18
19
20
21
22
23
24
25
26
27
28
29
30
over 30
What is your mother's highest degree of education?
Less than high school
High school graduate
Some college
Associate's Degree (2 yr)
Bachelor's Degree (4 yr)
Graduate degree (MBA, MA, MS, PhD, JD, MD, etc.)
What is your father's highest degree of education?
Less than high school
High school graduate
Some college
Associate's Degree (2 yr)
Bachelor's Degree (4 yr)
Graduate degree (MBA, MA, MS, PhD, JD, MD, etc.)
Figure 3. Student Opinion Survey
Student Survey
Econ ID # ______________
Please rate your level of agreement with the following statements by checking the appropriate box.
Statement
I prefer a “flipped” classroom format to a
“traditional” lecture format.
I prefer a traditional lecture format.
I would prefer a hybrid model where some topics
are still lectured on in class and others are flipped.
I believe that I learned more economics with this
class format.
I would prefer to take my next economics course
with this type of class format.
I enjoyed working in groups in class.
I learned a lot working in groups in class.
There is too much work to do outside of the class
for this course.
I worked more in this class than in my other
classes this semester.
The workload for this course was appropriate.
I read the textbook in this class more than I would
have in other economics classes.
I read the textbook in this class less than I would
have in other economics classes.
I read the textbook in this class the same as I
would in any economics class.
I had difficulty taking notes and paying attention to
lecture.
Other comments:
Strongly
Agree
Agree
Neither Agree
Nor Disagree
Disagree
Strongly
Disagree