Fifth International Developments
in Economics Education (DEE) Conference
Wales Millennium Centre, Cardiff, UK
9-10 September 2009
REGULAR REVISION STUPID!
- THE RETENTION OF KNOWLEDGE
AFTER THE END OF MODULE DELIVERY
Paulina K. Klich
Dr. Piotr M. Jaworski
School of Accounting Economics and Statistics
The Business School
p.jaworski@napier.ac.uk
Introduction
2
Introduction
• The study, which was financed by the Higher Education
Academy Economics Network within the framework
of 2008/09 Mini Project Programme.
• It aimed to measure effectiveness of IT-assisted continuous
assessment regime used for Economics 1 module delivered
to four cohorts of first year undergraduates’ students of NUBS
in two academic years 2006/07 and 2007/08.
• The study involved retesting a sample of 60 participants from
all cohorts with the same tests and measuring the retention of
knowledge delivered in the course of the module:
– multiple regression analysis confronted this retention level with
the amount of preparation devoted to particular tests in the first
week of material presentation and later.
– also relation between the preparation and the initial tests’ results
was researched.
3
The Module: Economics 1
• First year undergraduate one-trimester module
presenting
basic
economic
concepts
of micro- and macroeconomics consisting of ten
units of material
• Addressed to Business School students
as an introduction to more advanced Economics 2
module and to all other interested, who would like
to gain elementary economic knowledge.
• Average semester enrolment: 200 students.
• Every unit involved 2h lecture and 1 hour tutorial
4
Assessment Regime
• The assessment was divided into two parts:
an essay (40% of final mark) and two
IT-assisted tests constituting summative
assessment (60% of final mark).
• Another part of the assessment regime
were
weekly
10-question
quizzes,
which
served
as
a
“gateway”
to lecture notes.
5
The Tests and the Quizzes
• 2006/07
– Every test was worth 30% of the final mark;
– The questions were drawn out of 15 question
from 10 sets previously used for quizzes;
– The quizzes did not contribute towards the final.
• 2007/08
– Every test was worth 25% of the final mark;
• 0.75% for 20 questions from quizzes;
• 1% for 10 new true/false questions.
– 10% for all quizzes completion within one week of the
material presentation;
6
2006/07 vs. 2007/08
Tests and Quizzes
Quiz
Test
2006/07
2007/08
2006/07
2007/08
No of questions
From quizzes
new
10
N/a
N/a
10
N/a
N/a
30
30 for 1%
0
30
20 for 0.75%
10 for 1% each
Total value towards
the final mark
Drawn out of
0%
Max* 1%**
30%
25%
15
20
Unlimited
Unlimited
Pass mark
No of attempts
75
(15 per unit)
5 questions 5 questions
None
2
100
(20 per unit)
None
2
7
The Results
• Passing rate: out of 260 and 140 participants
respectively in the first and second trimesters of 2007/08
89.62% and 91.43% passed.
• This was a substantial increase compared
to 83.70% and 83.79% in 2006/07.
• On the other hand, the average final mark decreased from
64.61% and 61.13% for trim. 1 and 2 of 2006/07 to
56.70% and 54.10% in 2007/08.
• Also standard deviation of the final marks decreased.
8
The Research
• The structure introduced seems
to have been effective:
– the amount of revision effort was fairly considerable;
– systematic work was induced and strengthened by the usage of
the simple economic concepts of utility and discounting.
• Open issue: whether the expected
formative character of the assessment
regime materialised or we just landed with
another form of summative assessment.
9
Research Stages
• First stage:
– retesting sample of students with exactly the same tests
which they had done in the course of the module.
– counting the number of all quizzes completed in the first
and following weeks for every student.
• Second stage:
– correlate different parts of the assessment with the initial
tests results and the results for knowledge retention.
10
The Sample
• Invitation sent to 100 students resulted in 10% response:
only 10 students retested in the same supervised
environment and 50 voluntary completion of the retests
from home after the invitation sent to all 685 students.
• The sample mean of final mark was 69.41 with standard
deviation of 12.21 and median 70.50 (for the population
64.98 and 12.28 respectively).
• The sample dispersion similar to the population but the
mean and median around 5 points higher, what suggests
that the sample included better students.
• The 60 sample out of the population of 685 gave us
a confidence interval of a little bit more than 12% (95% level)
11
The Data
• Independent variables:
– Numbers of quiz completion done within the first week and later;
separately for material covered by test 1 and test 2 (NQF1, NQL1,
NQF2, NQL2).
– the fact whether the student was under the initial (0) or adjusted
regime (1) of assessment; (dummy variable NAS).
– Essay mark (ESS),
– Number of trimesters from the end of module delivery (NTR),
• Independent variable:
– for the initial tests results regression: tests results (TST1 and TST2)
– for the results for knowledge retention regression: retention of
knowledge (RTN1 and RTN2) which was constructed as the retest’
score (HEA1 and HEA2) divided by the results of the original test’s
scores (TST1 and TST2)
12
Initial Tests
Results Regression
After using several sets of dependent variables it turned out that the most
meaningful results are generated by using NQF, NQL and NAS as
dependent variables in the multiple linear regression with constant (CON).
NQF
NQL
NAS
CON
TST1
Coefficient Standard
Error
5.30
0.34
0.13
0.01
0.10
0.02
13.91
0.28
R square
Standard error of
estimate
Degrees of freedom
F statistic
t stat
15.51
15.30
4.39
49.22
51.33%
4.11
681
239.43
TST2
Coefficient Standard
Error
6.57
0.34
0.13
0.01
0.06
0.02
14.17
0.27
R square
Standard error of
estimate
Degrees of freedom
F statistic
t stat
19.35
13.76
2.88
51.72
54.12%
4.25
681
267.80
13
Knowledge Retention
Regression
The most meaningful results we obtained for the regressions
without constant (CON) involving NQF, NQL and NTR.
NQF
NQL
NTR
RTN1
Coefficient Standard t stat
Error
18.28
3.05
5.99
0.01
0.25
0.04
1.46
0.42
3.43
R square
74.10%
38.40
Standard error
of estimate
Degrees of freedom
57
F statistic
54.37
Coefficient
RTN2
Standard
Error
2.43
0.39
0.22
16.86
1.33
-0.08
R square
Standard error
of estimate
Degrees of freedom
F statistic
t stat
6.92
3.43
0.34
79.43%
30.43
51
65.65
14
Initial Tests
Results Regression
The results confirmed our belief that the regular
and continuous work on the material was crucial
for the level of final test mark:
– the number of quiz completion within first week of
material presentation had the highest influence on the
final mark in the case of both tests with very high
significance and high R square.
– Although the two other dependent variables, the
number of quiz completion after the first week and the
influence of the new system of assessment, have also
been statistically significant, their positive influence has
been much lower, if almost non-existent.
15
Knowledge Retention
Regression
The number of quizzes completed in the first week
turned out to be the most important factor:
– with relatively high significance it influenced the
retention of knowledge for both tests with the R
squares being even higher than for the “final mark”
regressions.
– This influence was also considerably higher than the
“final mark” influence.
– Completing quizzes later was insignificant and had
practically no influence on test 1 but a moderately
statistically significant modest influence on test 2.
16
Knowledge Retention
Regression
• In case of the NTR the results were
reversed: statistically significant for test 1
and insignificant for test 2.
• The time passing increased the retention
in a similar degree that practising after the
first week of module delivery.
• In the case of the test 2 it was slightly
negatively related to knowledge retention.
17
Knowledge Retention
Regression
• Test 2. results were a surprise; possible explanation:
– difference between materials covered by tests 1. (micro-) and 2.
(macroeconomics);
– micro aspects of the economy under constant revision in the
modules following Economics 1,
– macroeconomic issues are not so often recalled in the course of
later studies.
– Therefore, the micro revision in following modules serves as a
similar tool of revision as the quiz completion after the first week
of module completion in the case of macroeconomics.
18
Conclusions
• The results confirm the fact that revision within the first
week after material presentation is vital in the process
learning: in the time of module delivery it increases tests
results and after it enhances the knowledge retention.
• We hope that we managed to create a regime of
assessment for the students not to the students which
according to Biggs (1998) represents the distinction
between formative and summative assessment.
19
Conclusions
• Furthermore, the assessment regime led to expected by
us action, which was increased continuous and
systematic revision which also constitutes a condition for
formative assessment (Sadler, 1989).
• This, we hope, induced self assessment skills which
students should learn through their undergraduate
degrees that, according to Boud (2000) may foster
lifelong learning, which is our lecturers’ another
important goal.
20
Questions and Comments
are Welcomed!
21
REFERENCES
•
Biggs, J. (1999). Teaching for quality learning at university. Buckingham: Open University Press.
•
Biggs, John (1998). Assessment and Classroom Learning: a role for summative assessment?.
Assessment in Education: Principles, Policy & Practice, 5 (1), 103-110.
•
Boud, David (2000). Sustainable Assessment: rethinking assessment for the learning society.
Studies in Continuing Education, 22 (2), 151-167.
•
Brown, S., & Knight. P. (1994). Assessing learners in higher education. London: Kogan Page.
•
Conway, M.A., Cohen, G. & Stanhope, N. (1992). Why is it that university grades do not predict
very long-term retention? Journal of Experimental Psychology, 121(3), 382-384.
•
Deci E. L., & Ryan R. M. (1985). Intrinsic motivation and self-determination in human behaviour.
New York: Plenum.
•
Deci E L. & Ryan R. M. (2000). Intrinsic and Extrinsic Motivations: Classic Definitions and New
Directions. Contemporary Educational Psychology 25, 54-67.
•
Entwistle N. (2000). Promoting deep learning through teaching and assessment: conceptual
frameworks and educational contexts. Paper presented at TLRP Conference, Leicester,
November. http://www.tlrp.org/pub/acadpub/Entwistle2000.pdf
•
Gibbs, G. & Simpson, C. (2004). Does your assessment support your students’ learning? Journal
of Teaching and Learning in Higher Education, 1, 3–31.
•
Jaworski P. (2009) Let Us Teach the Way We Preach: Using Economic Concepts in IT-Assisted
Continuous Assessment Scheme for Introductory Economics Module, Manuscript submitted for
22
publication in Advances in Business Education.
REFERENCES
•
Jaworski P. (2008) Enhancing student learning through the introduction of a WebCT-based
continuous assessment scheme. Educational Research Paper Series no. 6, School of Accounting,
Economics & Statistics, Napier University Business School
•
Jeppson K., (2008) Increasing student examination rate by use of weekly home assignments.
European Workshop on Microelectronics Education
•
Kniveton, Bromley H. (1996). Student Perceptions of Assessment Methods. Assessment &
Evaluation in Higher Education, 21 (3), 229-237.
•
Marton F. & Saljo R. (1976). On qualitative differences in learning, outcome and process, British
Journal of Educational Psychology 46, 4-11.
•
Miller, C.M.I. & Parlett, M. (1974). Up to the mark: A study of the examination game. Guildford:
Society for Research into Higher Education.
•
Ramsden, P. (1992). Learning to teach in higher education. London: Routledge.
•
Snyder, B.R. (1971). The Hidden Curriculum. Cambridge, MA: MIT Press.
•
Sadler, D.R. (1989). Formative assessment and the design of instructional systems. Instructional
Science, 18. 119-144.
•
Sambell, K. & McDowell, L. (1998). The construction of the hidden curriculum: messages and
meanings in the assessment of student learning. Assessment and Evaluation in Higher Education,
23(4), 391-402.
23