Education 3504 Week 3
reliability & validity
observation techniques
checklists and rubrics
Reliability and Validity
• validity
– extent to which you are measuring what
you think you are measuring
• reliability
– extent to which information you are
gathering is consistent and unambiguous
Reliability
• chance factors
–
–
–
–
characteristics of student
characteristics of test / evaluation
conditions of administration
variations in scoring
Validity
• danger of thinking you are measuring
one thing, actually measuring another
– trivial knowledge
– lowest-common denominator
– regional knowledge/one culture/gender
Validity
• face validity
– appears important, relevant, interesting
• construct validity
– appears to capture intended quality or
behaviour - e.g. what does “good citizenship”
consist of?
validity and reliability are related
• but can vary independently
Checklists and Rubrics
checklist construction
• Checklist must be based on
objectives of the course, unit, or
lesson
• each item should be clear and precise
• each item should avoid generalities
and focus on specifics
checklist construction
• checklist should be designed for
feedback to students and parents
– so need to be understandable
– might want to distribute to students
before hand
• must do so if intended for grading
checklist construction
• if students can use them, can be used for peer and selfevaluation
• focus on a few specifics, so keep checklist short
– if list needs to be longer, focus on only small section at a time
• anecdotal summary comments can be tied to checklists
rating scales
5 -Extremely active
4 -somewhat active
3 -average active
2 -somewhat active
1 -extremely active
rating scales
Extremely Active (5 points)
• always out of seat
• rarely sits without squirming
• makes noise, yells a good deal of the time
Rubrics
scoring schemes
clear expectations
Generic
Essay
Subject
content
awareness
of issues
critical
thinking
Takes a
position
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken where
relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
a thoughtful statement
of position is presented
and defended through
logical arguments and
carefully selected
supportive detail; the
arguments presented
build to a consistent
conclusion
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
a position is adopted
and logically argued;
appropriate supporting
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
a position is adopted
demonstrated
detail is supplied
and logically argued
…but flexible
Generic
Essay
Subject
content
awareness
of issues
critical
thinking
Takes a
position
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken where
relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
a thoughtful statement
of position is presented
and defended through
logical arguments and
carefully selected
supportive detail; the
arguments presented
build to a consistent
conclusion
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
a position is adopted
and logically argued;
appropriate supporting
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
a position is adopted
demonstrated
detail is supplied
and logically argued
levels of mastery
Generic
Essay
Subject
content
awareness
of issues
critical
thinking
Takes a
position
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken
where relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
a thoughtful statement
of position is presented
and defended through
logical arguments and
carefully selected
supportive detail; the
arguments presented
build to a consistent
conclusion
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
a position is adopted
and logically argued;
appropriate supporting
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
a position is adopted
demonstrated
detail is supplied
and logically argued
multi-dimensional
Generic
Essay
Subject
content
awareness
of issues
critical
thinking
Takes a
position
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken where
relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
a thoughtful statement
of position is presented
and defended through
logical arguments and
carefully selected
supportive detail; the
arguments presented
build to a consistent
conclusion
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
a position is adopted
and logically argued;
appropriate supporting
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
a position is adopted
demonstrated
detail is supplied
and logically argued
parallelism
Generic
Essay
Subject
content
awareness
of issues
critical
thinking
Takes a
position
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken where
relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
a thoughtful statement
of position is presented
and defended through
logical arguments and
carefully selected
supportive detail; the
arguments presented
build to a consistent
conclusion
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
a position is adopted
and logically argued;
appropriate supporting
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
a position is adopted
demonstrated
detail is supplied
and logically argued
high but attainable standards:
cumulative criteria
Generic
Essay
Originality
Subject
content
awareness
of issues
critical
thinking
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
originality, insight, and
creativity are
demonstrated; the
paper goes beyond
repeating what others
have said and
contributes something
new to our
understanding of the
topic
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken where
relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
demonstrated
allowance for part marks
Generic
Essay
Subject
content
awareness
of issues
critical
thinking
Takes a
position
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken where
relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
a thoughtful statement
of position is presented
and defended through
logical arguments and
carefully selected
supportive detail; the
arguments presented
build to a consistent
conclusion
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
a position is adopted
and logically argued;
appropriate supporting
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
a position is adopted
demonstrated
detail is supplied
and logically argued
allowance for part marks
A - EXCELLENT (85 - 100) - A Markedly Exc eptiona l P erformance
¦
¦
¦
origin ality, insight , and creativit y are demons trated; the pap er goes beyond r epeating
wha t other s have said and contributes something n ew to our unde rstand ing of the
topic
a comprehens ive grasp of the subject matter is demons trated, including an in-depth
unde rstand ing of the relevant conc epts, theo ries, and issues related to the topic
addressed
an awarene ss of d iffering v iew-points is demons trated and a rigo rous assessment of
these unde rtaken whe re relevant
an abili ty to think criti cally is demons trated in the analysis , syn thesis and evaluation
of relevan t info rmation
a thought ful statement of positi on is presented and defended through logical
arguments and carefu ll y selected suppor tive detail; the argum ents presented build to a
consistent conc lusion
a clear, fluent, and concise style highli ghts a well -writt en, tightly a rgued , and
logicall y structured essay
a virtuall y flawle ss mastery of all aspects of gramm ar, structure, and style is
demonstrated
B - SUPERIOR (70 - 84) - Cle arly Above Ave rage Performance
¦
a thorough grasp of the subject matter is demons trated
an awarene ss of d iffering v iew-points is demons trated and an assessment of these
attempted whe re relevan t
the paper goe s beyond de scription to interpretation, analysis, synthes is and eva luation
a positi on is adopted and logi call y a rgued ; approp riate suppo rting de tail i s suppli ed
a clear style which comm unicates well (but may contain occ asiona l or mi nor flaws in
the mechan ics of spelli ng, gra mm ar, sentence structure, etc.) is evid ent in the logical
presentation of a reasonab le argument
C - SATI SFACT ORY (55 - 69) - A Fully Competent Paper
¦
¦
a basic grasp of the subject matter is demons trated
accura te information incorporating relevant sou rces and references is conveyed
a positi on is adopted and logi call y a rgued
an adequa te attempt at ana lysis, syn thesis, interpretation o r evaluation is evident
an acceptable style demonstrates an awareness of, and attention to, the p rinciples of
paragraph d evelopment, sentence struc ture, gramm ar and spelli ng, etc.
correspondence to real world
Generic
Essay
Subject
content
awareness
of issues
critical
thinking
Takes a
position
EXCELLENT
A (85 - 100)
A Markedly
Exceptional
Performance
a comprehensive grasp
of the subject matter is
demonstrated,
including an in-depth
understanding of the
relevant concepts,
theories, and issues
related to the topic
addressed
an awareness of
differing viewpoints is
demonstrated and a
rigorous assessment of
these undertaken where
relevant
an ability to think
critically is
demonstrated in the
analysis, synthesis and
evaluation of relevant
information
a thoughtful statement
of position is presented
and defended through
logical arguments and
carefully selected
supportive detail; the
arguments presented
build to a consistent
conclusion
SUPERIOR
B (70 - 84)
Clearly Above Average
Performance
a thorough grasp of the
subject matter is
demonstrated
an awareness of
differing viewpoints is
demonstrated and an
assessment of these
attempted where
relevant
the paper goes beyond
description to
interpretation, analysis,
synthesis and
evaluation
a position is adopted
and logically argued;
appropriate supporting
SATISFACTORY
C (55 - 69)
A Fully Competent
Paper
a basic grasp of the
subject matter is
asserts viewpoint
without acknowledging
alternative viewpoints
accurate information
incorporating relevant
sources and references
is conveyed
a position is adopted
demonstrated
detail is supplied
and logically argued
evolving
• changes if you discover papers that do not
fit criteria
– add descriptors that accurately describe what
you are seeing
evolving
• changes as your teaching changes
• changes as students change
• changes as environment changes
ease of use
A - EXCELLENT (85 - 100) - A Markedly Exc eptiona l P erformance
origin ality, insight , and creativit y are demons trated; the pap er goes beyond r epeating
wha t other s have said and contributes something n ew to our unde rstand ing of the
topic
a comprehens ive grasp of the subject matter is demons trated, including an in-depth
unde rstand ing of the relevant conc epts, theo ries, and issues related to the topic
addressed
an awarene ss of d iffering v iew-points is demons trated and a rigo rous assessment of
these unde rtaken whe re relevant
an abili ty to think criti cally is demons trated in the analysis , syn thesis and evaluation
of relevan t info rmation
a thought ful statement of positi on is presented and defended through logical
arguments and carefu ll y selected suppor tive detail; the argum ents presented build to a
consistent conc lusion
a clear, fluent, and concise style highli ghts a well -writt en, tightly a rgued , and
logicall y structured essay
a virtuall y flawle ss mastery of all aspects of gramm ar, structure, and style is
demonstrated
B - SUPERIOR (70 - 84) - Cle arly Above Ave rage Performance
a thorough grasp of the subject matter is demons trated
an awarene ss of d iffering v iew-points is demons trated and an assessment of these
attempted whe re relevan t
the paper goe s beyond de scription to interpretation, analysis, synthes is and eva luation
a positi on is adopted and logi call y a rgued ; approp riate suppo rting de tail i s suppli ed
a clear style which comm unicates well (but may contain occ asiona l or mi nor flaws in
the mechan ics of spelli ng, gra mm ar, sentence structure, etc.) is evid ent in the logical
presentation of a reasonab le argument
ease of use
Checklist - Space Community Unit
1 = weak
2 = fair
3 = good
4 = very good 5 = supe rior
Name:
KEY BEHAVIORS
Date:
A
Grade:
#1 Respond to
Stimuli
#2 Respect the
space of others
#3 Discipline Self
#4 Develop
appreciation
B. #1 Become
aware of Body and
Voice
Acts on experience
and information in a
strikingly coherent
way
Respects the space
of others every time
Readily takes action
to discipline self
Indicates wholehearted enthusiasm
Begins to use body
and voice in a rich
variety that reflects
personal
knowledge, feelings
and values
Act on experience
and information in a
consistently
coherent way
Respects the space
of others
consistently
Takes action to
discipline self
Indicates
enthusiasm
Begins to use body
and voice in a
variety of ways that
reflects personal
knowledge, feelings
and values
Acts on experience
and information in a
generally coherent
way
Respects space of
others most of the
time
Sometimes takes
action to discipline
self
Indicates interest
Begins to use body
and voice some
different ways that
reflect personal
knowledge, feelings
and values
Acts on experience
and information in a
somewhat
haphazard way
Respects the space
of others some of
the time
With help takes
action to discipline
self
Indicates
acceptance
Begins to use body
and voice with
encouragement
that reflects
personal
knowledge, feelings
and values
Acts on experience
and information
randomly;
sometimes rambles
Respects space of
others hardly ever
Even with help does
not always discipline
self
Indicates
impatience
Begins to use body
and voice with
difficulty even when
encouraged to
express personal
knowledge, feelings
The studen ts demons trate the abilit y to:
Superior
A
Social
1.
respond to stimuli
2.
respect the space of others
3.
discipli ne self
4.
deve lop appreciation o f the
work of self and others
1 2 3 4 5
Comments
5
Very Good
B
Physical
1.
become aware of body and
voice as instruments of
exp ression
C Intellectual/Emotional
4
Good
3
Fair
2
1.
comm unicate through space
2.
deve lop concen tration
3.
recogn ize form of drama
(beginning, midd le, end)
Weak
1
4.
deve lop and exercise
im agination
5.
exp lore emotion (role play)
rubrics workshop
• Objective:
Produce the best cookie possible
rubrics workshop
• Objective:
Produce the best cookie possible
• Was it difficult to come to a
consensus on criteria for “best”?
why we evaluate
•
•
•
•
motivation
administration
accountability
certification
motivation
• problems:
– test anxiety
Norm referenced
• describes student performance relative to
other students
– e.g.: ranks 11th in group of 30
Criterion referenced
• describes student performance compared
to a clearly defined criteria or task
– e.g.: “adds single-digit whole numbers”
motivation
• problems:
– overemphasis distorts learning process
• emphasis on external motivation
motivation
• problems:
– overemphasis distorts learning process
• not everything we teach can be assessed
(e.g., non-cognitive skills, attitudes, etc.)
administration
• general trend towards greater
emphasis on assessment
• general shift from selection to
accountability
accountability
• government to taxpayers
– 2003, K-12 = 3.8 billion dollars…
accountability
• teachers
– accountable for mandated curriculum
– accountable for student learning
– to oneself over time
accountability
• students to their parents
certification
• right of passage
• selection and
allocation
(gatekeeper)