Data Driven Decision-Making
Erin Baumgartner & Ava Howard
Biology Department
Western Oregon University
How do you get a group of passionate faculty
with diverse backgrounds to agree on
instructional practice?
DATA!
For Example…
• What’s better for students- many small or few large exams?
• What’s better for students to have the same lecture and lab
instructor or different instructors?
• What’s better for students – multiple choice or open-ended
assessments?
• How do we improve the laboratory experience?
• How consistent are exam practices between instructors?
Does it matter?
• Is there a difference between 4-item or 5-item response
choices on multiple choice questions?
• How are we serving non-traditional, underrepresented, and
other student groups?
Exam Size & Frequency
Hypothesis:
• More frequent, smaller exams will
improve student study habits and
performance and decrease anxiety.
Methods:
• Control-comparison
– Two midterms, 90 points, 50 minutes
– Four miniexams, 45 points, 25 minutes
• Quantitative:
– Embedded indicator questions on exams
– Student survey – Likert response scale
• Qualitative:
– Student comments
Results:
… But students love them
• Decrease anxiety – No
• Improved study habits – No
• Improved performance – Minimal
Conclusions:
Conclusions
Lecture and Lab Instructors
Hypothesis:
• Students with same lecture and lab
instructor gain more content knowledge
and improve attitudes about science.
Methods:
• Naturally occurring comparison groups
– Students with same lecture-lab
– Students with different lecture-lab
• Quantitative:
– Proportion of correct answers on concept inventory
– Student attitude survey – Likert response scale
– Two-sample t-test overall comparison
• MANOVA across demographic groups
Results:
Average Post-Course Content Score
p = 0.7481; t = 0.3215 w/265 df
1
Average Post-Course Attitude Score
p = 0.3453; t = 0.9456 w/237 df
0.6
5
0.4
4
0.2
0
same lecture-lab
different lecture-lab
Condition
Average Likert Score
Average percent score
0.8
3
2
1
0
same lecture-lab
different lecture-lab
Condition
Conclusions:
Conclusions
Types of Exam Questions
Hypothesis:
• Students perform better on open-ended
assessment items.
Multiple Choice:
Open Ended:
What is the role of the sphincters in the
digestive system?
What is a sphincter?
Identify a place in the body
where you might find one.
A. Secrete enzymes
B. Prevent food from moving too quickly
or moving backward through the
system
C. Keep food moving through the system
D. Adjust pH in different parts of the
system
Methods:
• Within group comparison
– Exams consisted of multiple choice and openended items (short-answer, fill-in-the-blank,
labeling, essay)
• Quantitative:
– Correlation of performance on item type against
overall performance
Results:
Exam item type score correlated with total exam score
100
90
Percent MC Score
Score on exam item type
80
70
Percent written score
60
Linear (Percent MC Score)
50
Pearson correlation coefficient = 0.855
40
Linear (Percent written score)
Pearson correlation coefficient = 0.900
30
20
10
0
0
20
40
60
Overall exam score
80
100
Conclusions:
Question Type
Points each # on exam total points % allocation notes
multiple choice
(MC)
2
16
32
53%
2 points per item
Matching
1
12
12
20%
1 point per item, combined into
appropriate number of questions
Fill in the blank or
label diagram
2
6
12
20%
can be combined into fewer
questions; or split- single word
answers can be worth 1 point,
while short (phrase or one
sentence) answers worth 2
Open Ended
(Short answer/
Drawing/ Essay)
4
1
4
7%
35
60
100%
TOTALS:
can be divided into multiple parts
or combined into one question
with several elements
Dissemination is Important
5 items?
4 items?
1.
1.
A carefully formulated scientific
explanation that is based on a
large accumulation of
observations and has been
tested and supported by
multiple scientists
a. Control
b. Theory
c. Hypothesis
d. Fact
e. Postulate
A carefully formulated scientific
explanation that is based on a
large accumulation of
observations and has been
tested and supported by
multiple scientists
a. Control
b. Theory
c. Hypothesis
d. Fact
•
Morrison, Susan and Free, Kathleen. (2001). Writing multiple-choice test items that promote and
measure critical thinking. Journal of Nursing Education 40, 17-24.
•
Rodriguez, M.C. (2005). Three Options are Optimal for Multiple-Choice Items: A Meta-Analysis of 80
Years of Research. Educational Measurement: Issues and Practice, 24(2), 3-13.
IRBs are Important
A Consistent Framework is Important
Thanks!
•
•
•
•
•
•
•
•
•
•
Lindsay Biga
Karen Bledsoe
Jim Dawson
Bryan Dutton
Emma Dutton
Amy Harwell
Kelly Kissane
Kristin Latham
Liz Martin
Jeffrey Snyder
Julie
Grammer