Avaluació experimental de la
millora de l’aprenentatge en
estadística
Pilar Muñoz, José-Antonio González, Erik Cobo, Lluis Jover
JORNADES D’INNOVACIÓ DOCENT A LA UPC: Presentació de
resultats dels projectes MQD
ICE 28/06/07
PRESENTACIÓ DE RESULTATS MQD
Outline
1. Aims and motivation
2. Model & subjects
3. Results
4. Conclusions and future work
2
PRESENTACIÓ DE RESULTATS MQD
Outline
1. Aims and Motivation
2. Model & subjects
3. Results
4. Conclusions and future work
3
PRESENTACIÓ DE RESULTATS MQD
Motivation
• To improve student learning
• To provide to students an automatic IT which generates
and solves individual exercises
• To apply statistical theory to formally measure its effects
• To employ real examples in teaching
4
PRESENTACIÓ DE RESULTATS MQD
Learning Model
5
PRESENTACIÓ DE RESULTATS MQD
Factors to be efficient in the teaching process
Training
Practice of methods and techniques with realistic cases
Instant feedback for students
Feedback for teachers
Immediate marking of work
Evaluation of knowledge gained
Evaluation of effort
To monitor progress
(globally or individually)
6
PRESENTACIÓ DE RESULTATS MQD
The tool: e-status (1)
• Learning by practicing:
– The exercise can be repeated
– Initial data are always different
• Immediate feedback providing:
– Right answer
– Error reason (if predicted)
• Students’ assessment:
– Any criteria (best answer, average, …)
• Students’ follow up:
– Both individual data and group summaries
• Broad range of problems: based on R
• Web-based tool
7
PRESENTACIÓ DE RESULTATS MQD
The tool: e-status (2)
More information on e-status: Gonzalez & Muñoz (2006)
(CAEE, 14(2): 151-159)
8
PRESENTACIÓ DE RESULTATS MQD
How e-status looks
Wording
Question
Answer
Grading
9
PRESENTACIÓ DE RESULTATS MQD
e-status previous experience
• Experience since 2003:
– Four schools: • Computer Science
• Maths and Stat
• Medicine
• Dentistry
– About 10 subjects
– More than 2000 students
– About 25000 executions
• High positive correlation between e-status use and
exam performance… but
…experience is not experiment
10
PRESENTACIÓ DE RESULTATS MQD
Outline
1. Aims and Motivation
2. Model & subjects
3. Results
4. Conclusions and future work
11
PRESENTACIÓ DE RESULTATS MQD
Setting
• Biostatistics course in the Dentistry School of the
University of Barcelona
• Prior experience in 2004/2005 academic course
• Duration of the stat subject: 35 hours
• Teachers:
– theoretical classes: 1
– lab groups: 3
• Experiment: 2005/2006 year (fall)
• Outcome: written final examination
12
PRESENTACIÓ DE RESULTATS MQD
Operation
13
PRESENTACIÓ DE RESULTATS MQD
Two “topics”
Stat course content was divided in two ‘balanced’ topics
A
B
• Descriptive statistics and
graphical representation
• Probabilities with Normal distribution
• Agreement
• Interval estimation of proportion and
mean
• Inference about one proportion
• Assessment of sample size
• Comparison of two means
• Inference about one mean
• Comparison of two proportions
• Goodness of fit chi-square
Every student had access to e-status
Students were randomly allocated to group 1 or 2
Each group only had access to e-status exercises on only one topic
Final exam contained questions about both topics A and B
14
PRESENTACIÓ DE RESULTATS MQD
Outcome
Exam
Problems in the practical exam
Topic
e11
e12
e21
e22
e31
e32
A
B
Score Yt A i
Score Yt B i
15
PRESENTACIÓ DE RESULTATS MQD
Participants & allocation
• All students (N =121) enrolled in the course
• Random assignment to 1 or 2 balanced with respect to Lab
group and new/old profile
• Teacher was not involved in randomization neither data analysis
• Final exam evaluator was masked to allocation group
16
PRESENTACIÓ DE RESULTATS MQD
Hypothesis
Yt j i stands for the exam performance:
• by student i
• assigned to intervention, t=1, 2
• in topic j=A, B
Hypothesis:
If e-status is effective, students in group 1 (2)
trained with e-status exercises in topic A (B)
should get better exam results in topic A (B)
17
PRESENTACIÓ DE RESULTATS MQD
Statistical model
Ytji = µ + t + πj + Φi + ij
•
•
•
•
t: fixed effect of intervention t=1, 2
πj: fixed effect of exam topic j=A, B
Φi: random effect of student i
ij: measure error assessing performance in student i for
question j
Assumptions:
• Access to e-status topic A(B) has no effect on exam topic B(A).
• Error independence between students
18
PRESENTACIÓ DE RESULTATS MQD
Statistical analysis
Let D1 i (D2 i) be the difference of scores in part A and B for the
student i receiving intervention 1(2):
D1i = Y1Ai – Y1Bi = (µ + τ1 + πA + Φi + iA)-(µ + πB + Φi + iB) = τ1 + πA - πB+ iA- iB
D2i = Y2Ai – Y2Bi = (µ + πA + Φi + iA)-(µ + τ2 + πB + Φi + iB) = - τ2 +πA -πB+ iA- iB
As
E(D1i)= τ1 + πA - πB
E(D2i)= -τ2 + πA - πB
V(Dji)= V(iA- iB) = 2σ2ε
Then
ED1  D2   1  2
VD1  D2  2  n  2  n  4  n
2
2
2
50 students per group provide 80% power to highlight an effect equal to
0.7x σ (α=0.05)
19
PRESENTACIÓ DE RESULTATS MQD
Outline
1. Aims and Motivation
2. Model & subjects
3. Results
4. Conclusions and future work
20
PRESENTACIÓ DE RESULTATS MQD
Main results
21
PRESENTACIÓ DE RESULTATS MQD
Linear Mixed-Effects Model
The random model, fitted with R, replicated the results
SΦ = 2.81 (CI95%: 2.43 to 3.24)
Sε = 1.48 (CI95%: 1.31 to 1.68)
3
2
2
Quantiles of standard normal
Standardized residuals
1
0
1
0
-1
-1
-2
-2
-3
2
4
Fitted values
6
8
-2
-1
0
1
2
Standardized residuals
22
PRESENTACIÓ DE RESULTATS MQD
Outline
1. Aims and Motivation
2. Model & subjects
3. Results
4. Conclusions and future work
23
PRESENTACIÓ DE RESULTATS MQD
Conclusions
+1. It is feasible to evaluate interventions in teaching with formal
experiments
 Random allocation
 Masked evaluation
 Without interfering in course development
+2. e-status improves student exam performance
In an exam over 10 points, e-status improves
performance by

0.96 points (CI95%: 0.20, 1.72)
24
PRESENTACIÓ DE RESULTATS MQD
Interpretation (1)
What did the students do when they had no access to e-status?
That is, what is the reference for the intervention?
a) Did they not study statistics at all?
b) Did they spend their time on another kinds of exercises?
If a, the estimated e-status effect is mediated by an increase in the
time spent by the student to study.
If b, e-status increases learning efficiency, since it improves the
amount of learning with respect to the alternative method.
25
PRESENTACIÓ DE RESULTATS MQD
Interpretation (3)
If e-status influences advanced capabilities, such as motivation or
statistical reasoning, there may be some contamination between
interventions A and B: that is, intervention 1 (2) employing e-status
on set A (B) would have some learning effect τ‘1 (τ‘2) on set B (A),
and then
E(D1  D2 ) = (τ1 + τ2 ) – (τ‘1 + τ‘2 ) = (τ1 - τ‘1) + (τ2 - τ‘2 )
If so, this design estimates e-status direct effect minus delayed,
cross-over, effects
If τ‘ positive, this design underestimates overall effects on learning
26
PRESENTACIÓ DE RESULTATS MQD
Future work
1. Repeat the randomized experiment many times in many
courses
You are invited to do it!
2. Also you are invited:
- to use it in your teaching work
http://key.upc.es/estatus
3. To share your ideas with us
pilar.munyoz@upc.edu, jose.a.gonzalez@upc.edu,
erik.cobo@upc.edu
27
PRESENTACIÓ DE RESULTATS MQD
Thanks for your attention
Q&A
28