Penny Craig, Ruth Jones, & Lisa Pufpaff,
Department of Special Education

Examined inter-rater agreement pre
and post rater training
◦ 10 faculty raters
◦ 3 different assessments
◦ 2 exemplars for each assessment
◦ Inter-rater agreement very poor
BSU Assessment Forum April 2015
2


Online, self-guided rater training with
expanded rubrics and narrated PPTs
◦ Little change in inter-rater agreement
◦ Social validity among raters very high
Recently published:
◦ Pufpaff, L. A., Clarke, L., & Jones, R. E.
(2015). The effects of rater training on
inter-rater agreement. Mid-Western
Educational Researcher, 27(2), 117-141.
http://www.mwera.org/MWER/
BSU Assessment Forum April 2015
3






Selected one assessment from previous
summer
3 faculty and 2 course instructors
Baseline inter-rater agreement on 5
exemplars
Discussed differences and revised rubric
Repeated inter-rater agreement on 5
different exemplars
Discussed differences and revised rubric
BSU Assessment Forum April 2015
4
Unsatisfactory
Does not establish the value
of the paper topics as regards
the legal, theoretical, and
historical bases of special
education practices.
Basic
Establishes the relationship of
the paper topic(s) to the
identification and support of
students with ELN. Describes
legal foundations and past
practices contributing to the
establishment of
requirements for special
education identification and
services.
Proficient
Meets Basic requirements and
presents the models,
theories, philosophies, and
research methods that form
the basis for special
education practice, including
FAPE, LRE, the continuum of
placement options, and the
provision of related services
such as specialized health
care.
Distinguished
Meets Basic and Proficient
requirements and describes
how students are identified
with disabilities and the
educational needs they
present, including
considerations for culturally
and linguistically diverse
students.
Statutory law, eight examples
of case law, or corresponding
administrative regulations are
not presented or are not
relevant to topic.
Presents current statutory
law, a minimum of eight
court cases (total), and
current requirements as per
state regulations
relevant to the paper topics
Meets Basic requirements and
explains relationship of case
law to statute(s) and
administrative regulations.
Meets Basic and Proficient
requirements and cites court
cases specifically with a
summary of plaintiff and
defendant positions, judicial
ruling, and relevance to topic
explained. Clarifies
progression of the litigation
through the court system.
Omits or minimizes the
rights, roles, and
responsibilities of families in
collaboration with schools to
provide meaningful access to
educational opportunity for
students with ELN.
Describes the effects of law
on student, parents, and
school staff and the role of
the family with the school,
citing a minimum of five
scholarly (peer-reviewed)
journals.
Meets Basic requirements and
describes the rights,
responsibilities, and
outcomes as per topic,
including the impact of the
dominant culture and
potential differences between
school and home.
Meets Basic and Proficient
requirements and provides
examples of best practice
recommendations for
disability area for support to
students with ELN, including
laws, policies, and ethical
principles of behavior
management and
reinforcement and models
and theories of deviance and
behavior problems.
5
BSU Assessment Forum April 2015


Completed this process 5 times before
reaching >90% agreement
This process resulted in a “rubric checklist”
to be used by both students and instructors
BSU Assessment Forum April 2015
6
Proficient
(Meets Basic)
Unsatisfactory
Basic
________Correctly identifies the legal
foundations of IDEA or Indiana’s
Article 7.
________Fails to completely or
correctly address the six components
of IDEA.
Correctly identifies the legal
foundations of:
________Constitution
________IDEA
________Article 7
________Correctly lists social and legal
history
________Correctly addresses six
components of IDEA:
___IEP
___FAPE
___LRE
___Eval process
___Parent involvement
___Procedural safeguards
_______Correctly addresses:
___Continuum of placements
___Related services
________Describes at least 2 court
cases
________Relates the cases to correct
key component(s) of IDEA
________Correctly describes one court
case
________Relates the case to incorrect
components of IDEA
________Fails to correctly address the
role of the family in regards to
discipline
Comment pertaining to all cells in
this rubric row: Simple mention of a
concept or topic does not meet
criteria. Be sure that topic is
“Correctly identified…correctly
explained and correctly discussed…”
________Correctly identifies and
defines the role of the family in
regards to:
________discipline
________procedural safeguards
________resolving conflicts
________right of the family to be
involved
Distinguished
(Meets Basic and Proficient)
Correctly explains legal foundations
from:
________Constitution
________IDEA
________Article 7
________social and legal history
Correctly explains:
________IEP
________FAPE
________LRE
________Eval process
________Parent involvement
________Procedural safeguards
________RtI
Synthesizes and interrelates
corresponding ideas relative to:
________how SWD are identified
________RtI and traditional model
________specific example of
educational needs of a SWD related to
law
________needs of culturally and
linguistically diverse individuals
________Correctly relates the 2 court
cases to specific regulation(s) within
Article 7
________Court cases are correctly
explained as they progress through
court system:
________plaintiff
________defendant
________final ruling
________progression through the
court system
________Correctly discusses relevance
of the court cases to both:
________IDEA
________Article 7
________Correctly discusses
components of discipline including all
of the following:
________FBA
________BIP
________Relates the provision of
discipline and procedural safeguards
to both:
________IDEA
7
BSU ________Article
Assessment Forum
April 2015
________Correctly discusses
components of discipline including all
of the following:
________manifestation det.
________suspension
________expulsion
________Relates discipline and
procedural safeguards to a court case
or peer-reviewed article
7



Digital Portfolio is a required assessment of
all teacher education candidates at 4 points
in their program
Normally assessed by course instructor
SPCE created a Portfolio Assessment
Committee made up of faculty
BSU Assessment Forum April 2015
8


Revised the Unit rubric into a rubric checklist
with increased specificity of performancebased verbs and quantifiers at each
performance level
Began creating departmental policies and
procedures for uniformity in administration
and assessment of portfolios across all SPCE
majors
BSU Assessment Forum April 2015
9
Rater training among PAC members
 Randomly divided 117 portfolios
among three PAC members for
evaluation
 27 portfolios evaluated by a second
rater
 Have not done statistical analysis yet

BSU Assessment Forum April 2015
10



Randomly divided 65 portfolios across 4 PAC
members
All 4 members assessed 13 (20%) portfolios
Have not run statistical data yet
BSU Assessment Forum April 2015
11
Evaluator
#1
Evaluator
#2
Evaluator
#3
Evaluator
#4
D
P
D
D
Rationale
P
P
P
P
Design
D
P
D
D
Environmen
t
Mechanics
D
P
D
D
P
D
D
P
Professiona
lism
Artifact
D
D
P
D
D
D
P
D
Summary
D
P
P
D
Student
A.H.
Reflection
BSU Assessment Forum April 2015
12
Evaluator
#1
Evaluator
#2
Evaluator
#3
Evaluator
#4
D
D
D
D
Rationale
P
P
P
D
Design
D
D
D
D
Environmen
t
Mechanics
D
P
D
D
P
P
P
D
Professional
ism
Artifact
D
D
P
D
D
D
D
D
Summary
D
D
D
D
Student
E.M.
Reflection
BSU Assessment Forum April 2015
13
Evaluator #1 Evaluator #2 Evaluator #3 Evaluator #4
Student B.T.
Reflection
P
P
P
D
Rationale
D
P
D
D
Design
D
D
D
D
Environment
P
D
D
D
Mechanics
P
P
P
P
Professionali
sm
Artifact
P
D
P
D
D
D
P
D
Summary
P
P
P
D
BSU Assessment Forum April 2015
14
Evaluator
#1
Evaluator
#2
Evaluator
#3
Evaluator
#4
Reflection
P
B
P
P
Rationale
B
B
B
B
Design
P
B
B
P
Environmen
t
Mechanics
P
D
P
D
B
P
D
P
Professional
ism
Artifact
P
P
P
P
P
D
P
D
Summary
P
P
P
P
Student E.S.
BSU Assessment Forum April 2015
15






Decision Point 1
http://nmalexander.iweb.bsu.edu/portfolio
Decision Point 2
http://herose.iweb.bsu.edu/portfolio
Decision Point 3
http://cmedwards3.iweb.bsu.edu/portfolio
BSU Assessment Forum April 2015
16