WHO WE ARE
https://www.youtube.com/watch?v=usQem53fDH8
A BIT MORE BACKGROUND
http://www.albion.edu/academics/departments/education
http://www.facebook.com/Albion.College.Teacher.Education
Approximately 1300 total enrollment; 80 in TEP
TEAC accredited through October 1, 2016
A BIT MORE BACKGROUND (cont’d)
Elementary certifications: English language arts, integrated
science, mathematics and social studies
Secondary certifications: biology, chemistry, earth/space science
(geology), English, French, German, history, mathematics,
physics, political science, psychology, social studies (major
only) and Spanish
K-12 certifications: French, German, Spanish and music
WHY HOLISTIC SCORING?
● Teacher Education Accreditation Council (TEAC)
accreditation weakness
● change from Teacher Education Accreditation Council
(TEAC) national standards to Council for the Accreditation of
Educator Preparation (CAEP) standards
● need to identify what would constitute evidence and a tool
for assessing the evidence
● financial constraints
● requires collaboration
HISTORY OF HOLISTIC SCORING
● Gray, J. (1982). Properties of writing tasks: A study of
alternative procedures for holistic writing assessment. final
report. http://eric.ed.gov/?id=ED230576
● Hunter, D. M. (1996). The use of holistic versus analytic
scoring for large-scale assessment of
writing.http://eric.ed.gov/?id=EJ543798
WHAT IS HOLISTIC SCORING?
Holistic scoring is a process for assessing and grading the
summative quality … based on professional and personal
knowledge, the particular process and event, the purposes at
hand, and the meanings shared by the individuals involved. An
overall impression includes everything...; this cannot be broken
into component parts, per se. A holistic system for scoring is
legitimate and reasonable because it requires the scorers to cast
a comprehensive view .... The process is synergistic because it
requires iterative thinking by each participant, and invokes
collaboration. It is a kind of gestalt; the whole is greater than the
sum of its parts. (after Dr. Judy Nichols Mitchell)
WHAT IS HOLISTIC SCORING (cont’d)?
Advantages—
Focuses on what writers can do, and actually did, rather
than on what they didn’t do;
Links function, content, and form-the acts of composing and
producing;
Serves as an evaluation measure and/or needs assessment;
and,
Is authentic to the demands of real writing tasks.
WHAT IS HOLISTIC SCORING (cont’d)?
Preparation—
Compose/select
prompt
Training—
Describe system
Collect materials
Explain prompt &
anchor trial
Select anchor set
Tally/discuss scores
Application—
Read and score all
materials
Sort/tabulate all
scores
Discuss and react
Select scoring scale Develop/confirm
criteria
WHAT IS HOLISTIC SCORING (cont’d)?
WHAT IS HOLISTIC SCORING (cont’d)?
Reference for four-point scoring scale—
Three categories (unsatisfactory/low; basic/middle; and
proficient/high);
Distinguished category eliminated due to DFFT/EI
conceptual framework for this category;
Round down on split scores (e.g., 0+1 or 2+3);
Non-adjacent and non-equivalent scores (e.g., 1, 3; 2, 4)
require third scoring; and,
Across three scores, count two lowest scores.
See also Danielson Framework for Teaching/Evaluation Instrument.
WHAT IS HOLISTIC SCORING (cont’d)?
Scoring Scale—
0 (←unsatisfactory)
0, 1 (←unsatisfactory)
(←proficient)
1, 2 (←basic)
3, 4 (←proficient)
2, 3, 4, 5 (←basic)
6, 7, 8
WHAT IS HOLISTIC SCORING (cont’d)?
SAMPLE ANCHOR SET SCORING TABLE
4
A
B
C
3
2
1
0
WHAT IS HOLISTIC SCORING (cont’d)?
WHAT DID WE ACHIEVE?
● TEP Curriculum Map
● Danielson Framework for Teaching Evaluation Instrument
● CAEP Standard 5
TEP Curriculum Map
https://www.dropbox.com/s/q4q5lazfnktzoon/Converted%20TE
P%20%26%20Danielson%20FFT%20Evaluation%20Instrumen
t_correlation_v2.1.xlsx?dl=0
CAEP Standard 5:
Provider Quality Assurance & Continuous Improvement
The provider maintains a quality assurance system comprised of
valid data from multiple measures, including evidence of
candidates’ and completers’ positive impact on P-12 student
learning and development. The provider supports continuous
improvement that is sustained and evidence-based, and that
evaluates the effectiveness of its completers. The provider uses
the results of inquiry and data collection to establish priorities,
enhance program elements and capacity, and test innovations to
improve completers’ impact on P-12 student learning and
development.
CAEP Standard 5:
Provider Quality Assurance & Continuous Improvement
(cont’d)
Quality and Strategic Evaluation
5.1 The provider’s quality assurance system is comprised of
multiple measures that can monitor candidate progress,
completer achievements, and provider operational effectiveness.
Evidence demonstrates that the provider satisfies all CAEP
standards.
5.2 The provider’s quality assurance system relies on relevant,
verifiable, representative, cumulative and actionable measures,
and produces empirical evidence that interpretations of data are
valid and consistent.
CAEP Standard 5:
Provider Quality Assurance & Continuous Improvement
(cont’d)
Continuous Improvement
5.3. The provider regularly and systematically assesses
performance against its goals and relevant standards, tracks
results over time, tests innovations and the effects of selection
criteria on subsequent progress and completion, and uses results
to improve program elements and processes...
WHAT DID WE ACHIEVE (cont’d)?
● Field Experience & Student Teaching Feedback Forms
● Lesson Plan Form
● reflection (e.g., Time I Heard Self)
FIELD EXPERIENCE FEEDBACK FORM
Old Form
com/s/nv8yc7s72ob86p2/ASSESSMENT%20for%20NEW%20PRO
GRAM%20updated%20Summer%202007.doc?dl=0
New Form
https://www.dropbox.com/s/jr1adctce07a5oa/field%20experie
nce_final%20evaluation__tier%201%5BR%5D.docx?dl=0
STUDENT TEACHING FEEDBACK FORM
Old Form
https://www.dropbox.com/s/hd297ic7jqc2gk7/finalfeedback%20form%20for%20mentor%20teachers.docx?dl=0
New Form
https://www.dropbox.com/s/82mlhomshig2si7/field%20experie
nce_final%20evaluation%5Bmentor_teachers%5D__tier%204
.docx?dl=0
WHAT PROBLEMS DID WE HAVE?
● Mental shift from scoring on separate attributes to scoring
artifact as a whole.
● Review one artifact in relation to one DFFT element; even if
two might be applicable, cumbersome to score.
● Explain the context of the artifact—course, assignment, and
objectives of assignment.
● As we progressed in our understanding of the DFFT, we reassigned artifacts to different elements.
● Can’t do it all—small program, so we can score artifacts
from all students without sampling; we can’t score all
artifacts from all students. Choose for coverage of the
framework.
WHAT’S NEXT?
● Add more “tiers” to our process.
● Deepen our understanding of the DFFT to improve the
validity of artifacts-to-elements match.
● Expand to actual pre-service teacher performance.