COLLECTING AND ANALYZING DATA:
MEASURING STUDENT SUCCESS
Rebecca Orr, Ph.D.
Professor of Biology
Agenda
• Efforts made to improve student success:
• Online homework using MasteringBiology
• Peer tutoring
• Online quizzing
•
• Strategies for data collection:
• Preparing to begin
• Modifying as collection progresses
• Involving your students
• Case study: Impact of pre-exam quizzing on student
outcomes
• Lessons learned
Student Success Challenges
• Have you ever heard:
• “I studied so hard, and it just didn’t
work!”
• My question:
• How can student success be improved
in the introductory biology (majors)
course?
My Evolution of Mastering Use
• Began using Mastering to create assignments
and made them available for optional student
use.
• Student feedback indicated a trend:
• Successful students were using Mastering
• Course evaluation surveys by students said:
“Make Mastering mandatory!”
• Less successful students did not tend to utilize
Mastering
• Moved to making Mastering mandatory by
assigning a portion of lecture credit (10%) to
Mastering.
Is There a Significant, Positive Correlation Between
Completing MasteringBiology and Exam Scores?
• Students required to complete MasteringBiology
homework assignments (10% lecture grade).
• Overall exam average was compared to percentage
MasteringBiology completed by end of semester.
• Spearman's Rank Order correlation was run:
(rs(138) = 0.571, P = .000)
• A statistically significant, positive correlation between
percentage of MasteringBiology homework completed
and exam average was found.
Results
• Students completing more work in
MasteringBiology were more successful on
exams.
Problem: How to Increase Student Success in
Introductory (Majors) Biology?
• My Strategy, Spring 2011 semester:
• Evaluate the efficacy of providing peer tutors vs. requiring
MasteringBiology homework (using exam scores to measure)
• Offered optional, online pre-exam quizzes to all using
MasteringBiology
• Results:
• Peer tutoring appears to have no benefit over offering
required MasteringBiology homework.
• Significant, positive correlation between participating in
optional quizzing and increased exam score observed.
Cognitive Science of Learning
• Introduction of Desirable Difficulties:
“Conditions of instruction that appear to create
difficulties for the learner, slowing the rate of
apparent learning, often optimize long-term
retention and transfer.”
Testing Serves as Learning Events
• Testing as a learning vehicle vs. an assessment
vehicle.
• Evidence shows: Practice of recalling information
is more powerful than re-presentation of
information.
• Reality is: Students re-present information to
themselves (i.e., reading a chapter over and over
again, underlining things, etc.), focusing on
storage rather than retrieval.
Can An Online Homework Platform Be Used
For Quizzing to Increase Student Success?
• Introduced required pre-exam quizzing delivered
via MasteringBiology
• Students required to take pre-exam quizzes during
the Fall 2011 semester (n=199) and Spring 2012
semester (n=174)
• My Question:
Does taking pre-exam quizzes using
Mastering result in significantly higher exam
scores?
Preparation
• Before beginning, set up spreadsheet to collect data
necessary to answer questions.
• My predesigned columns included:
• Name (deleted prior to sending to statistician)
• Exam # and grade
• Number of pre-exam quizzes taken
• Grade on each pre-exam quiz
• Set up color coding to designate “drops” and “not in my course”
• Data gathered by exporting grade book from Mastering.
Student Participation Compliance in
Required Pre-exam Quizzing
Fall 2011 & Spring 2012
% Students Completing Quizzes prior to
Exam
% Quizzes Exam 1
Completed n= 362
Exam 2
Exam 3
Exam 4
n= 339
n= 321
n= 305
100% 78.5
59.9
62.9
59.3
67% 16.3
22.4
*
*
50% *
*
26.5
25.9
33% 3.0
11.5
*
*
0% 2.2
7.4
10.3
14.8
Comparison of Exam Averages
100% Quiz Takers to 0% Quiz Takers
Fall 2011
Spring 2012
Comparison of Exam Averages by Percentage
of Quiz Taking: Spring 2012
Analysis by Groups
• Scores on Exams 1 and 2 were averaged, and
students were grouped by these averages.
• Groups Established:
• Pass: 70%+
• Fail: <70%
• Low (“weak student”): <60%
• Middle (“average student”): 60%-80%
• High (“strong student”): >80%
• Student scores were compared by group for 100%
quiz takers vs. 0% quiz takers.
Performance of Passing vs. Failing Students:
100% Quiz Takers vs. 0% Quiz Takers
Exam 3
Exam 4
Performance by Low, Middle and High Students:
100% Quiz Takers vs. 0% Quiz Takers
Exam 3
Exam 4
Data Analysis Often Generates New
Questions!
• Can students’ compliance with pre-exam quiz
requirement be increased?
• If percent pre-exam quiz compliance
increases, will significant gains in student
success in the course be observed?
Impact of Data Collection on Course
Approach
• Tell the students about the data!
• Ask them to predict! Who earns higher exam scores:
• Quiz takers or non quiz takers?
• Quiz takers that fail their quizzes or non quiz takers?
• Show them the data, and ASK:
• “What group do you want to be in?!”
The blue group? Or the red group?!
Pre-exam Quiz Participation Rate Increased
Significantly in Fall 2012 vs. Spring 2012
85
80
75
70
65
60
55
50
45
40
35
30
25
20
15
10
5
0
Exam 1
Exam 2
Exam 3
Exam 4
Spring: 100%
Participation
Fall: 100%
Participation
Spring: 0%
Participation
Fall: 0%
Participation
Percentage of ABC vs. DFW Grades Earned
in Spring 2012 vs. Fall 2012
70.00%
60.00%
50.00%
40.00%
Spring
Fall
30.00%
20.00%
10.00%
0.00%
ABC
DFW
Study Results
• Student participation in pre-exam quizzing results in
significantly higher exam grades.
• Benefit realized to student is based on participation
(rather than performance) in quizzing.
• Participation in pre-exam quizzing results in significantly
higher exam scores for students of diverse abilities.
• Stronger students may see slightly greater benefits from
pre-exam quizzing (as evidenced by exam averages).
• Significant increase in student compliance with required
quizzing was accompanied by significantly higher
percentage of ABC rates vs. DFW rates.
Student Perception of Mastering:
Homework and Pre-Exam Quizzing
100
90
80
70
60
50
40
30
20
10
0
Mastering
Mastering
Quizzes
Quizzing
Homework
Homework
Increased
Results Led to
Helped Me
Increased
Understanding + Studying
Keep up with Understanding
Material
Plan to
continue
quizzing in
future
Study Conclusions
• Pre-exam quizzing using online platform seems to
provide a viable testing environment for student
learning.
• Using online platform allows instructors more time to
teach!
• All students showed positive gains in exam scores as
a result of participating in pre-exam quizzing.
Lessons learned
• Begin process of gaining IRB approval right away.
• Consult with your statistician about what you want to
investigate before you start.
• Speak with colleagues about your investigation to get a
feel for questions they may raise.
• If introducing a new teaching method or technology,
consider giving a student survey to gather their thoughts
and comments.
Lessons learned (cont.)
• Design your data collection tools before beginning the
study.
• Gather and organize more than you think you need!
• Stay current in updating master spreadsheet throughout
the semester:
• Falling behind can become overwhelming!
• New questions often arise as you review the data.
• Modifications in what you record may be necessary. Much
easier to do this “as you go” than to go back and wade
through the entire semester’s worth of data!
• Tell students about your study- involve them in the
process when possible.
Benefits of Studying Student Success Data
• Allows you to identify what you are doing that is
truly effective.
• Sharing results with students motivates them and
helps them to feel that your requirements will yield
positive gains for them.
• Provides intellectual challenge to you that provides
additional purpose and interest!
Acknowledgements
• Shellene Foster
Professor of Mathematics, Collin College
• Pushpa Ramakrishna, Ed.D.
Professor of Biology, Chandler-Gilbert Community College
References
• 1. Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way:
Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M.
Hough, & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating
fundamental contributions to society (pp. 56-64). New York: Worth Publishers.
• 2. Halamish, V., & Bjork, R. A. (2011). When does testing enhance retention? A distributionbased interpretation of retrieval as a memory modifier. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 37, 801-812.