Andis Klegeris
Biology, UBCO
andis.klegeris@ubc.ca
1
Objectives:
 Flexible teaching vs. flexible learning
 Innovation in teaching: why is it needed?
 Student skills
 Move from innovation to scholarship
 Describe strategies for assessing student:
 perception
 problem-solving skills
 learning of course concepts and content
 Future directions
2
Flexible teaching vs.
flexible learning
Innovation in teaching:
why is it needed?
• The goal of flexible learning is to enhance the
educational experience of our students through
imaginative use of new educational technologies in
ways which permit more active learning by students,
while freeing time in medium and large classes to be
used for purposes other than content delivery.
• Common types of flexible learning environments
include online, flipped, and blended…
4
Potentially measurable outcomes
of teaching innovation
 Higher TEQ scores
 Increased student satisfaction/perception
 Improved learning of course contents
 Improved student skills
 Free up time to do other things
5
Perception vs. effectiveness
6
Potentially measurable outcomes
of teaching innovation
 Higher TEQ scores
 Increased student satisfaction/perception
 Improved learning of course contents
 Improved student skills
 Free up time to do other things
7
Student skills
 Program-specific skills
Business program
(Bovinet, 2007)
Mathematics program
(Hussain and Adni, 2011)
Oral communication
Numerical calculations
Interpersonal skills
Spatial visualization
Ability to work in a team
Algebraic manipulations
Research skills
Data analysis
Decision making
Estimation
8
Student skills
 Generic competencies (Tuning methodology)
 Systemic
 abilities and skills concerning whole systems (combination of
understanding, sensibility and knowledge)
 Interpersonal
 social interaction and co-operation
 Instrumental
 problem solving, cognitive and methodological abilities
9
Student skills (BIOC 308)
Pearson’s correlations of students’ marks obtained by various assessment
techniques ; BIOC 308, 2010, N = 42
* Correlation significant at P<0.05 level; ** Correlation significant at P<0.01 level
Klegeris, A., Bahniwal, M., Hurren, H. (2012) Lack of correlation between distinct
university student skill sets identified by using a panel of assessments: a two year study. Int.
J. Arts Sci. 5:479-498.
10
Move from innovation to
scholarship
•
Assess impact of educational interventions
(innovations)
• Qualitative data
• Quantitative data
11
Perception (satisfaction)
Specific skills (e.g., problem
solving skills)
Learning of course content
12
Why such a study is needed?
 Problem-based learning (PBL) in a small group
setting has a robust positive effect on student
learning and skills as well as their engagement and
satisfaction with the process.
 Very little research has been done on the
educational benefits of PBL in a large classroom
setting.
 Several studies have suggested that PBL may not
be superior to conventional educational approaches
in all aspects of learning.
13
Why such a studies are needed?
 It cannot be assumed that introducing the PBL
technique (or any other new teaching method) to
a large undergraduate class setting will
automatically lead to enhanced student learning as
well as satisfaction.
 The superiority, or at least the non-inferiority, of
PBL over the standard course delivery techniques
must be proven for each individual PBL delivery
method.
 DO NO HARM!!!
14
Perception (satisfaction)
Problem solving skills
Learning of course content
15
Perception of PBL process:
anecdotal evidence (informal surveys)
 Question: Would you like to continue with PBL,
studying another case this term? Why?
YES = 41 out of 44 students
 Summary of positive comments:
Process fun/enjoyable/interesting (28)
Process makes you think (17)
Retain/learn the information better (15)
Enjoy group interactions (13)
Learn practical information(11)
Enjoy researching (10)
Prefer this method of learning over others (8)
16
Perception of PBL process:
anecdotal evidence (informal surveys)
 Question: Would you like to continue with PBL,
studying another case this term? Why?
NO = 10 out of 44 students
 Summary of negative comments:
Discomfort with the unknown (student hypotheses &
research rather than the professor providing with the
answers) (6)
Exam discomfort (4)
The extent of the information that needs to be researched is
overwhelming (2)
17
Perception of PBL process:
formal surveys (moving from anecdote to evidence)
 Formal study conducted with BIOC 308/309
students during 2010/2011 academic year
 Study approved by UBCO Human Research Ethics
Board
 Students were asked to complete a survey posted
on WebCT Vista
 44 out of 59 students participated in the study
during the first term
 28 out of 45 students participated in the study
during the second term
18
Participating in
Problem Based
Learning has:
Perception
b
1 - strongly disagree
2 – disagree
3 - neither agree
or disagree
4 – agree
5 - strongly agree
Klegeris, A., Hurren
H. (2011) Impact of
problem-based
learning in a large
classroom setting:
Student perception
and problem-solving
skills. Adv. Physiol.
Edu. 35:408-415
Attending
classes with
traditional
lecture formats
has:
P
value,
sign
test
1
a
2
3
4
5
1
2
3
4
5
1
Increased my motivation to
participate in class
0
3
10
14
7
2
13
13
5
1
0.001
2
Not increased my motivation to
attend class
12
8
6
4
4
0
8
16
9
1
0.017
3
Enhanced my communication skills
0
2
12
19
1
5
21
5
3
0
<0.001
4
Increased my motivation to do
well in the course
0
3
12
15
4
1
3
17
11
2
0.48
5
Enhanced my retention of
course content
1
2
5
15
11
1
8
11
13
1
0.002
6
Not increased my understanding of
course content
6
18
7
3
0
2
18
10
3
1
0.049
7
Assisted my learning in other
courses
0
6
12
14
2
1
5
11
16
1
0.63
8
Increased my comfort level in
working in groups
0
0
7
16
11
9
I like the idea of evaluating
myself and my group members
2
5
7
17
3
10
If given a choice, I would
choose courses that used
Problem Based Learning over
traditional lecture format
0
4
5
19
6
c
19
Perception of PBL process:
anecdotal evidence
 How many PBL cases would you like to see next
term?
0 (no PBL)
1
2
3
Dec 2009
Dec 2010
(42 respondents)
(38 respondents)
4%
19%
37%
39%
5%
16%
53%
26%
(2 students)
20
Perception of PBL process:
student participation
Term 1
(60 students)
21
Perception of PBL process:
student participation
Term 1
(60 students)
Term 2
(44 students)
22
Perception (satisfaction)
Problem solving skills
Learning of course content
23
Subject-specific problem-solving skills
 At the beginning of the Fall 2010/11 term students were asked to complete a
pharmacology-related problem-solving exercise not related to the course
material they were about to study
 They were informed that they would be asked to complete a similar exercise in
December
 After two PBL cases were conducted during the term, students were actually
asked to solve not a similar problem, but the same problem they were given at
the beginning of the term
 The results were marked by an undergraduate TA and also by a contents expert
in a blinded manner
24
Problem-solving skills:
assessed by a TA
25
Problem-Solving Skills:
Assessed by a course content expert
26
Problems with the problemsolving skills exercise:
 Students were writing the same test twice
 This approach was used to allow pre- and post-exposure
comparison by Student’s t-test for paired observations
 There was no control group
 Class could be randomly split with one half having PBL
exercises included, but the other half exposed to lectures only
with instructor crossover – additional funding (or a volunteer to
teach a course for free) is needed!
 Content interference
 Problems were related to the subject of the course
 Bias in marking
 Undergraduate Teaching Assistant (TA) was marking the tests
as they became available; instructor was marking blinded
27
Generic problem-solving tests:
 Test general problem solving skills not related
to pharmacological or biological topics
 In this case a different group of students taking
other course could be used as controls
 Crossover design by using two comparable
problem sets
28
Generic problem-solving tests:
Controlled, blinded studies, crossover design, 2 tests
Test A
Test B
Test A
Test B
Test B
Test A
vs.
PBL 1
PBL 2
Test B
Test A
Klegeris, A., Bahniwal, M., Hurren, H. Improvement in generic problemsolving abilities of students by use of tutor-less problem based learning in a large
classroom setting. CBE Life Sci. Educ. (in press )
29
Generic problem-solving tests:
Controlled, blinded studies, crossover design, 2 tests
10
Beginning of Term
9
Test A
Test B
PBL 1
PBL 2
Average mark (/13)
8
7
6
5
4
3
Beginning of Term
2
1
Test B
Test A
0
Klegeris, A., Bahniwal, M., Hurren, H. Improvement in generic problemsolving abilities of students by use of tutor-less problem based learning in a large
classroom setting. CBE Life Sci. Educ. (in press )
30
Generic problem-solving tests:
Controlled, blinded studies, crossover design, 2 tests
12
Test A
Test B
PBL 1
PBL 2
Test B
Average mark (/13)
10
*
8
6
Beginning of Term
4
End of Term
2
Test A
0
Klegeris, A., Bahniwal, M., Hurren, H. Improvement in generic problemsolving abilities of students by use of tutor-less problem based learning in a large
classroom setting. CBE Life Sci. Educ. (in press )
31
Generic problem-solving tests:
Studies with internal control, crossover design, 3 tests
Test A
Test B
Test C
Test B
Test C
Test A
PBL x 2
Test C
Test A
Test B
32
Generic problem-solving tests:
Studies with internal control, crossover design, 3 tests
P = 0.06; N = 53
Test A
Test B
Test C
Test B
Test C
Test A
PBL x 2
Test C
Test A
Test B
PBL interventions
33
Generic problem-solving tests:
Studies with internal control, crossover design, 3 tests
P = 0.06; N = 53
Test A
Test B
Test C
Test B
Test C
Test A
PBL x 2
Test C
Test A
Test B
13% increase in previous study
5.7% increase in this study
½ term  shorter exposure time?
PBL interventions
34
Perception (satisfaction)
Problem solving skills
Learning of course content
35
Learning of course content
Large 1st year biology class with 2 sections
instructed by different professors using mainly
didactic lectures (232 and 268 students)
PBL
Midterm exam (3 identical questions)
PBL
Final exam (3 identical questions)
36
Learning of course content
Grades
Section 1
MT2 Overall
MT2 Q1
Sect. 2 PBL MT2 Q2
MT2 Q3
MT2 3Q
average
Final Overall
Final Q1
Final Q2
Sect. 1 PBL Final Q3
Final 3Q
average
63.068 MT2 Overall
82.19 MT2 Q1
Section 2
Difference
67.43
87.5
-4.362
-5.31
45.45 MT2 Q2
66.28 MT2 Q3
MT2 3Q
64.64 average
61.2
71.15
-15.75
-4.87
75.28
-10.64
62.39 Final Overall
42.53 Final Q1
44.02 Final Q2
57.46
35.71
23.94
4.93
6.82
20.08
50.37 Final Q3
Final 3Q
45.64 average
31.51
18.86
30.39
15.25
PBL
effect?!!!
6.278
10.32
Above
Expected
Future plans
Identify other courses/techniques that help improve student problemsolving skills
We have an original assessment tool available for anybody
interested
Assessment of specific courses dealing with various problems (math,
genetics)
Campus –wide assessment
Could be looking for collaborators from all faculties at UBC
Okanagan
38
Acknowledgements
• UBCO BIOC 308, BIOC 309 and other students
participating in these studies
• H. Hurren, M. Bahniwal, L. Spielman
• Drs. J. Pither, J. Loeppky, R. Lekivetz
• Dr. P. Arthur, J. Hirtz (UBCO Centre for Teaching and Learning)
• Drs. J. Boon and J. Kluftinger (UBCO Biology)
• UBCO I.K. Barber School of Arts and Sciences
Curricular Innovation Award
• UBCO Grant for Integration of Teaching and Research
in Undergraduate Education
39
andis.klegeris@ubc.ca
https://people.ok.ubc.ca/andislab/teaching/index.php
40