St. Anthony’s School
16 Jan 2013
1
Background information
2
Enhance the effectiveness of
learning and teaching
How to start
Curriculum
Teaching
Learning
Assessment
What
to start
3
Evaluation after Assessment
Are the
questions
too difficult
or too easy?
What are the
follow-up
actions?
Why did
students choose
A, not B, C or D?
Have the
students learned?
Have they
studied hard?
4
5
Data – Good to start
Student MCI
Item MCI
Cronbach Alpha
Graph shape
6
2 Modified Caution Indexes
Student Modified
Caution Index
Students’
Performance
Item Modified Caution
Index
Question
items’
Performance
7
2012-13 P.4
1st Reading
Exam
Student MCIxP Chart
MCI vs Performance of Student
1.0
21
0.9
Student Performance
39
0.8
40
0.7
46
68
0.6
23
65
0.5
44
0.4
45
0.3
48
53
0.2
69
0.1
17
42
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Student Modified Caution Index
0.7
0.8
0.9
1.0
56
8
Student MCIxP Chart
2012-13 P.4
1st Reading
Exam
9
More focused strategies
Type A
Type B
The teaching and learning are suitable
for them.
Encourage them to keep on.
Know the difficult questions
Miss the simple questions
Careless mistakes
Don’t rely on luck
Type D
Type C
Adjust the teaching content
Raise their learning motivation
Need more consolidation
Answering techniques
Careless mistakes
Focus on the foundation
Don’t rely on luck
10
2012-13 P.4
1st Reading
Exam
Item MCI xP Chart
MCI vs P-value of Item
1.0
Q9
0.9
Q6
0.8
Q18
0.7
Q21
Item P-value
Q19
0.6
Q16
Q12
0.5
Q24
0.4
Q17
0.3
Q13
Q14
0.2
Q1
0.1
Q5
Q23
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Item Modified Caution Index
0.7
0.8
0.9
1.0
Q25
11
Item MCI xP Chart
MCI vs P-value of Item
1.0
Q9
0.9
Q6
0.8
Need
Q18
0.7
improvement
Q21
Item P-value
Q19
0.6
Q16
Q12
0.5
Q24
0.4
Difficult
questions
0.3
0.2
Q17
Problem
Q13
questions
Q14
Q1
0.1
Q5
Q23
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Item Modified Caution Index
0.7
0.8
0.9
1.0
Q25
12
MCI item P-value chart
Item Type
A
1112 P.3 Final
Exam
20
1213 P.4 First
Exam
17
B
1
2
C
1
0
D
3
7
Total no. of
25
26
According the Item Modified Caution Index, most of the items set are normal.
13
Cronbach Alpha (Reliability)
Cronbach
Alpha
0 – 0.3
0.7-0.9
Reliability
> 0.9
high
low
acceptable
14
2011-2012 P.3 First Exam
Blue line
Students
Red line
Items
Purple line
Average
Slightly
Easy
15
2012-2013 P.4 First Exam
Blue line
Students
Red line
Items
Purple line
Average
Appropria
te
16
Which question
items should be
improved?
17
Which question
items are
difficult to our
students?
18
SP Xpress analysis on ‘2011-2012
Primary 3 Third Exam Paper’
 SP Xpress analysis results:
 Type A: 20 items
 Type B: 1 item
 Type C: 1 item
 Type D: 3 items
 Let’s examine one item from each category (Type B, C,
D)
19
Exam item example: Type B
On which floor of the centre, can Grandpa find
presents for Jack?
A. 1/F B. 2/F C. 3/F D.4/F
20
Possible reasons for item being Type B
 Type B: Item is not difficult but requires amendments
because many students couldn’t answer it.
 Possible reasons:
 Item design flaws
 Not close to what teacher taught in class
 Students lack specific training in reading comic strips
21
Teacher examined the Type B item
 Students found it difficult because …
 Need to process and connect many ideas all at the same
time
Question
Comic strip
Which floor?
1/F,2/F,3/F,4/Ffloor directory
… of the Centre …
Star Shopping Centre
… can Grandpa find
presents …
We can buy all the gifts …
presents?
presentsrobotstoys(4/F)
… can Grandpa …
‘We’ refers to ‘grandparents’
22
Recommended solutions
 Increase students’ chances in reading English comic
strips in classroom teaching or leisure reading
 Teacher needs to illustrate more clearly in connecting
ideas
 Teacher needs to teach more explicitly which means
more visual demonstrations
23
Exam item example: Type C
In which step does Jack’s mum add
‘soy sauce’ and ‘wasabi’ to the sushi?
A. Step 1
B. Step 3
C. Step 6
D. Step 10
24
Possible reasons for item being Type C
 Type C: Item is difficult and requires amendments
because many students couldn’t answer it.
 Possible reasons:
 Confusing description
 Question misleading students
 Question containing error(s)
25
Teacher examined the Type C item
 Students did not have the relevant personal experience in
making and/or eating sushi
 The recipe did not mention dipping sushi into ‘soy sauce’ or
‘wasabi’ after the finish
 Without experience, students could not interpret the correct
answer which is the final step
 This question contains the error which requires outside
knowledge uncommon among the students
26
Exam
item
example:
Type
D
After cutting the vegetables and crab meat, Jack’s mum ____.
C.takes some rice wine vinegar and mixes it with the sushi rice
This is the answer but the
option rephrased the
instructions.
27
Features of Type D item
 Type D: Item is difficult and doesn’t require
amendments. It is suitable for screening the high
ability students. Low and mid- ability students can’t
answer it.
 Type D items should be kept at a low proportion in an
assessment.
 The rephrasing is difficult for students to interpret the
correct answer.
28
Analysis of TSA P.3 Reading Paper
 Predict P.3 students’ results in the upcoming TSA reading
assessment
 Compare the overall students’ ability from different years
 Examine the neglected elements in our teaching
 Assumption: TSA assessment covers the basic competency that
all students are supposed to have acquired by the end of a
learning stage
29
SP Xpress analysis on ‘2011 TSA P.3
Reading Paper (3ER1)’
2011-2012 P.3
2012-2013 P.3
Mean
19.28
18.95
SD
4.78
4.76
Type A
18
20
Type B
6
3
Type C
0
1
Type D
2
2
30
An Interesting Fact: A common
Type B item across 2 years
 Part 1B Question #1
How many chapters are there in this book?
 ‘Chapter / Table of Contents’ can be an obstacle
 Students never encounter ‘Chapter’ in storybook teaching
(replacing textbook teaching)
31
An Interesting Fact: A common
Type B item across 2 years
 Sam buys two sandwiches with the Rainbow Fast Food coupon. How
many sandwiches can he get?
C. three sandwiches (= Buy 2 + 1 Free)
 Students are not familiar with ‘Buy 2 Get 1 Free’ concept even though
they can understand every single word
32
An Interesting Fact: A common
Type D item across 2 years
 “… too. In the end, John wins the race. All the people cheer
loudly …”
 Read lines 11-12. What do the people do when John wins the
race?
C. shout happily
 Students are not familiar with synonyms
cheer loudly
shout happily
33
Personal reflection
Teachers become aware of the quality of the
assessments
1.

Accumulate professional experience in setting better
assessments
Teachers prepared for the unexpected and
unlearned
2.

Teachers become cautious and adjust their teaching
34
Things to do
 Suggest Ts reflect on their exam paper using
SP Xpress
 Facilitate Ts process the data with technical
supports
 Set up a school based question bank
 Nurture a culture on giving a more constructive
feedback to both Ts and Ss after assessment
35
Thank you so much
for your time!