Life after levels
Daisy Christodoulou
Research & Development
Manager, Ark Schools
Life after levels
How can we use the latest assessment research
to help us design replacements for national
curriculum levels?
• What was wrong with levels?
• Possible replacements 1: using questions
• Possible replacements 2: using
comparative judgment
Illusion of a common language
‘Can compare two fractions to identify which is larger’
Which is bigger: 3/7 or 5/7?
90% get this right
Which is bigger: 3/4 or 4/5?
75% get this right
Which is bigger: 5/7 or 5/9?
15% get this right
Qtd in Wiliam, Principled Assessment Design, SSAT 2014
Illusion of a common language part 2
Understand and use ratios, proportions and percents in
a variety of situations.
1. Identify 50% of 20.
2. Identify 67% of 81
3. Shawn got 7 correct answers out of 10 possible answers
on his science test. What percent of questions did he get
correct?
4. J.J Redick was on pace to set an NCAA record in career
free throw percentage. Leading into the NCAA
tournament in 2004, he made 97 of 104 free throw
attempts. What percentage of free throws did he make?
Paul Bambrick-Santoyo, Driven by Data
5. J.J Redick was on pace to set an NCAA record in career free throw
percentage. Leading into the NCAA tournament in 2004, he made 97 of 104
free throw attempts. In his first tournament game, Redick missed his first
five free throws. How far did his percentage drop from before the
tournament game to right after missing those free throws?
6. J.J Redick and Chris Paul were competing for the best free-throw shooting
percentage. Redick made 94% of his first 103 shots, while Paul made 47 out
of 51 shots.
a) Which one had a better shooting percentage?
b) In the next game, Redick made only 2 of 10 shots while Paul made 7 of
10 shots.W hat are their new overall shooting percentages.
c) Who is the better shooter?
d) Jason argued that if Paul and J.J each made their next ten shots, their
shooting percentages would go up the same amount. Is this true? Why
or why not?
Paul Bambrick-Santoyo, Driven by Data
Can ask and
answer
questions
Pupil A
Pupil B
Pupil C
Can make
inferences when
reading texts
Can spell most
words accurately
So what could really replace
levels?
• Define criteria in terms of
– questions
– pupil work
What is ¾
+ 2/3?
What is the verb
in the following
sentence? I run to
the shops.
Fill in the blank
with the correct
word: Look over
______
(there/their/they’re
)
Pupil A
Pupil B
Pupil C
Research on multiple choice
questions
• “Well-designed selected-response items can probe
student understanding in some depth.” (Wiliam 2014)
• “Achieving “proper construction” of such tests— which
requires that incorrect alternatives be plausible, but not
so plausible that they are unfair—is, however, a
challenge.” (Bjork 2012)
• “Questions with more than one correct answer
considerably reduce the chances that students will get
the correct answer by guessing, and also allows the
teacher to provide a degree of challenge to higher
achieving students.” (Wiliam 2014)
Which option equals ½?
A. 50%
B. 25/100
C. 0.2
D. 25%
E. 20%
Which options equal ½?
A. 50%
B. 25/100
C. 0.2
D. 25/50
E. 20%
Which 2 options equal ½?
A. 50%
B. 25/100
C. 0.2
D. 25/50
E. 20%
Chances of guessing if there are 5
options
Options
Chances
If pupils don’t know the number of
correct answers
If pupils know there is 1 correct answer
1 in 32
If pupils know there are 2 correct
answers
If pupils know there are 3 correct
answers
If pupils know there are 4 correct
answers
1 in 10
1 in 5
1 in 10
1 in 5
Which characteristic of “Elegy Written in a
Country Churchyard” can be seen as Romantic?
A. It celebrates the supernatural.
B. It is written in iambic pentameter.
C. It emphasizes reason over emotion.
D. It deals with the lives of common people.
British Columbia leaving exam
Which is an example of a cause-andeffect relationship?
A. the Spanish Civil War caused the intervention of
British troops
B. the Manchurian Crisis caused the Washington Naval
Agreement
C. the Great Depression caused increased German
support for the Nazi Party
D. the Abyssinian Crisis caused formation of a League of
Nations military force
British Columbia leaving exam
How did the Soviet totalitarian system under
Stalin differ from that of Hitler and Mussolini?
A. It built up armed forces.
B. It took away human rights.
C. It made trade unions illegal.
D. It abolished private land ownership.
British Columbia leaving exam
If the value of another unit of a good in its next
best use is greater than the opportunity cost of
producing another unit of the good:
A. in a free market, the production of another unit
of this good would be considered economically
inefficient.
B. in a free market, at least one more unit of the
good will be produced.
C. under central planning, this information easily
gathered.
D. under central planning, the amount of the good
produced would decrease.
Macmillan Publishing, Learning Curve
Leaders of a miners’ union on strike against Coalco are contemplating
additional measures to pressure the company to accept the union’s
contract proposal. The union leaders are considering as their principal
new tactic a consumer boycott against Gasco gas stations, which are
owned by Energy Incorporated, the same corporation that owns Coalco.
The answer to which of the following questions is LEAST
directly relevant to the union leaders’ consideration of
whether attempting a boycott of Gasco will lead to acceptance
of their contract proposal?
A.
B.
C.
D.
E.
GMAT
Would revenue losses by Gasco seriously affect Energy
Incorporated?
Can current Gasco customers easily obtain gasoline elsewhere?
Have other miners’ unions won contracts similar to the one
proposed by this union?
Have other unions that have employed a similar tactic achieved
their goals with it?
Do other corporations that own coal companies also own gas
stations?
One inlet pipe fills an empty tank in 5 hours. A
second inlet pipe fills the same tank in 3 hours.
If both pipes are used together, how long will it
take to fill 2/3 of the tank?
A.
B.
C.
D.
E.
GMAT
8/15 hour
3/4 hour
5/4 hour
15/8 hour
8/3 hour
What is 20% of 300?
a)
b)
c)
d)
60
15
30
6000
Q1
Q2
Q3
Pupil A
A
B
A
Pupil B
A
A
B
Pupil C
D
A
C
What about essays?
Normally, we ask:
does this essay
meet the criteria?
Instead, we should
ask, is this essay better
than this essay?
Human judgment is comparative
and relative, not absolute
(1) Stealing a towel
from a hotel
(2) Keeping a dime you
find on the ground
(3) Poisoning a barking
dog
(1”) Testifying falsely
for pay
(2”) Using guns on
striking workers
(3”) Poisoning a
barking dog
Mozer, Michael C., et al. "Decontaminating human judgments by
removing sequential dependencies." Advances in Neural
Information Processing Systems 23 (2010).
Comparative judgment with No
More Marking
Comparative judgment in Maths – free event at the RI on
12th October http://rigb.org/about/news/summer2015/improving-mathematics-with-comparativejudgement
Conclusion
• Don’t replace levels with rehashed levels!
• Define criteria in terms of a) questions and
b) pupil work.
• Remember that human judgment is
comparative, not absolute.