CHAPTER 3
SCORING &
MODERATION
Strategies for
scoring
Reliability
Scoring
moderation
STRATEGIES FOR SCORING*
 The question format, options for how the student answers the
question, and the way answers are scored should all measure
the full range of complexity and depth of knowledge required
in the learning target.
 Together, the question format, the options for how the student
answers the question, and the way answers are scored provide
a reasonable estimation of student mastery of the learning
target.
 The connections between questions and their scoring criteria
should be apparent to anyone familiar with the learning
target.
 The rigor of scoring should increase in proportion to the
stakes attached to the assessment and complexity of the item
* As modified from Oregon Assessment Guideline (2014, p. 7)
SCORING FOR DIFFERENT ITEM T YPES
Scoring is an activity to assign a value that
can represent the degree of correct response
to an assessment task
 Multiple-choice or Ordered Multiple-choice items
 Easy scoring; objective
 Short constructed response
 More time consuming to score, should have clear and objective
scoring rubrics
 Extended constructed response
 Less objective, should have clear and objective scoring rubrics,
more time consuming to score
RELIABLE SCORING
The degree to which two or more scorers assign the
same score to a single student response
 One type of reliability
 Component of Assessment Quality
 To be discussed in Chapter 6
 Assessed by the use of two or more raters
 Use of scoring rubrics
 Can improve reliability since scores will be given consistently
to a particular item
 Improve validity as a score is specifically given to one or a set
of intended learning outcomes
ENSURING RELIABLE SCORING
 Selection of a set of responses
 Random samples of student work or
 Deliberately select from the high-, medium- and low-achieving
groups of students
 Score by multiple people
 Check for exact matches, difference of one score, difference
of two or more
 Discuss the results
 Evaluation of the effectiveness of scoring rubrics
 Training for additional scoring
 Evaluation of inter-rater reliability
OTHER ISSUES IN SCORING PROCESS
 Exemplar responses
 Responses that provide an excellent example of a particular
scoring level
 Can be used to assist in the scoring process
 Borderline responses
 Even the best rubrics will result in some borderline responses
 Assign a score, perhaps with a +/-
 What to do with missing responses
 Not-reached response
 Skipped response
 Can code as 0, or simply leave out (e.g. not use in percentage
calculations)
RIGOR OF SCORING
 Four main options for scoring constructed response
items, in order of ascending rigor:
1. Teachers directly score their own students’ work
2. Blind-scoring
 Teachers score randomized and anonymous selection of their own
student work
3. Multi-rater
 A selection of student work is scored by two different raters
4. Single or double scoring by third party
 Scoring is done by a separate, objective third party
 Most often related to high-stakes or standardized testing
SCORING MODERATION
 Activities:
 Ask two or more teachers to score the same portion of student
work (for example, 25% of the total number of student work)
 A leader conducts a discussion of mismatched scores
 Discuss reasons for mismatched score to evaluate own scoring
process
 Come to consensus
 Aims:
 To increase reliability
 Provide feedback for improving scoring rubrics
 For example, student responses in a single level may nevertheless
look qualitatively different – perhaps needing more than one level
 To promote professional development
BIBLIOGRAPHY
 E be l , R. L. , & Fri s bie , D . A . ( 1 9 91). E s s ent ials o f e duc a t i onal m e a suremen t .
E n g lewoo d Cl if f s, N J: P re n t i c e - Hall.
 N i t ko , A . J. , & B ro o k har t , S. ( 2 0 07). E duc a t i onal a s s essmen t o f s t ude n t s . Uppe r
Sa ddl e Ri ver, N J: Pe a r s on E duc a t i o n, In c .
 M c M i llan, J. H . ( 2 0 07). C l a s s ro om a s s e s s m e nt. P ri n c i pl e s a n d p r a c ti ce fo r
e f fe c t i ve s t a nd ard -b a s e d i n s t ruct ion ( 4 t h e d. ) . B o s to n: Pe a r s on - A l l y n & B a c o n .
 Ore g o n D e pa r t m e nt o f E duc a t i o n. ( 2 014, Jun e ) . As s essment g ui da n ce.
 Po ph a m , W. J. ( 2 014 ). Cri te rion -re ferenc ed m e a sureme nt: A h a l f - c ent ur y wa s te d?
Pa pe r pre s e nte d a t t h e An n ua l M e et i ng o f N a t i o nal Co un c il o n M e a sure ment i n
E duca ti o n, P h i la dephia , PA .
 Po ph a m , W. J. ( 2 014 ). C l a s s ro om a s s e s s m e nt: W h a t te a c h e r s n e e d s to k n ow . Sa n
Fra n c i sco, CA: Pe a r son
 Rus s ell, M . K . , & Ai ra sian , P. W. ( 2 01 2). C l a s s ro om a s s e s s m e nt: C o n c e pts a n d
a p p l i cati ons . N ew Yo rk , N Y: M c G raw - H ill .
 Steve n s , D . & Lev i , A . ( 2 0 0 5 ). I n t ro ducti on to ru b ri c s . As a s s e s s m e nt to o l to s ave
gr a d i n g t i m e , c o nvey e f fe c t i ve fe e d b a c k , a n d p ro m ote s t u d e nt l e a rn i n g . Ste rl i ng:
St y l us P ubl i shing, LLC
 Wi h a rdini , D . ( 2 010). As s essment deve lo pment II. Un publ ish ed m a n usc ript .
Re s e arc h a n d D eve lopment D e pa r t m e nt , B i n us B us i ness Sc h o o l, Ja ka r t a ,
In do n esia.
 Wi l son, M . ( 2 0 0 5). Co n st ruc t i ng m e asure s: An i te m re s po n se m o de lin g a ppro a c h .
N ew Yo rk : P s yc h ology P re s s , Tay l o r & Fra n c i s G ro up.
 Wi l son, M . , & Sl o a n e, K . ( 2 0 0 0). Fro m pri n c i ples to pra c t i c e : An e m be dde d
a s s essment s y s te m. Ap p l i e d M e a s u re m e nt i n E d u c ati on, 1 3 ( 2 ) , pp. 1 81 - 208 .
CREATIVE COMMONS LICENSE
S c o r in g a n d M o d e r a t i o n P P T b y t h e O r e g o n D e p a r t me n t o f E d uc a t i o n a n d B e r ke l ey
E v a l ua t io n a n d A s s e s s m e n t Re s e a r c h C e n te r i s l i c e n s e d u n d e r a C C BY 4 . 0 .
You are free to:
 Share — copy and redistribute the material in any medium or format
 Adapt — remix, transform, and build upon the material
Under the following terms:
 Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes
were made. You may do so in any reasonable manner, but not in any way that suggests the licensor
endorses you or your use.
 NonCommercial — You may not use the material for commercial purposes.
 ShareAlike — If you remix, transform, or build upon the material, you must distribute your
contributions under the same license as the original.
O r e go n D e p a r t me n t o f E d u c a t i o n w e l c o m e s e d i t i n g o f t h e s e r e s o u r c e s a n d w o u l d
g r e a t l y a p p r e c i a te b e i n g a b l e to l e a r n f r o m t h e c h a n g e s m a d e . To s h a r e a n e d i te d
ve r s i o n o f t h i s r e s o u r c e , p l e a s e c o n t a c t C r i s te n M c L e a n , c r i s te n .m c l e a n @ s t a te . o r. u s .