Teacher Performance
Assessment Consortium
November 10, 2010
Andrea Whittaker and Nicole Merino
Stanford University
1
Three Sponsoring
Organizations
•
AACTE – overall project management,
communication with programs
•
Stanford University – assessment
development and technical support
•
Council of Chief State School Officers
– policy development and support,
communication with state education
agencies
© Stanford Center for Assessment, Learning, and Equity, 2010
2
TPAC State Participation
Requirements
•
Sign off from state education agencies
•
Participation of preparation programs
•
Up to 3 funded through grant
•
Alignment and policy studies
•
Adaptation of TPA to address future state program
and policy needs
© Stanford Center for Assessment, Learning, and Equity, 2010
3
4
Lineage of TPAC
•
•
•
National Board for Professional Teaching Standards
(NBPTS) portfolio assessments – accomplished
teachers
Connecticut BEST assessment – teachers at end of
induction
Performance Assessment for California Teachers
(PACT) – pre-service teachers
source: SCALE @ Stanford University 5
© Stanford Center for Assessment, Learning, and Equity, 2010
APPLE Criteria
The National Board for Professional Teaching Standards
Administratively feasible
Publicly credible
Professionally acceptable
Legally defensible
Economically affordable
© Stanford Center for Assessment, Learning, and Equity, 2010
6
Design Principles:
Educative Assessment
•
•
•
•
•
Discipline Specific
Integrated Assessment
Student Centered
Analytic Feedback and Support
Represents a Complex View of
Teaching: Multiple Measures
© Stanford Center for Assessment, Learning, and Equity, 2010
7
The TPAC Assessment
System
Assessments Embedded in Local Programs
— examples —
The Capstone
Assessment
TPAC Assessment
Demonstrates :
Child
Case
Studies
Analyses of
Student
Learning
Curriculum
/Teaching
Analyses
• Planning Instruction
and Assessment
• Engaging Students
and Supporting
Learning
Observation/Supervisory Evaluation & Feedback
•Assessing Student
Learning
‣Reflecting
‣Academic Language
8
TPAC Assessment
Records of Practice*
Instructional and Social Context
Learning Segment of 3 to 5 Hours/Days
Planning
•Lesson Plans
•Handouts,
overheads,
student work
•Lesson
Commentary
Engagement
•Video clip(s)
•Teaching
Commentary
Assessment
•Analysis of
Whole Class
Assessment
•Analysis of
learning of 2
students
•Feedback to 2
students
•Next steps in
instruction
• Evidence of Academic Language Development
• Daily Reflections and Reflective Commentaries
© Stanford Center for Assessment, Learning, and Equity, 2010
9
Guiding Questions and
Analytic Rubrics
•
PLANNING
‣ Planning Focused, Sequenced
•
Instruction
‣
‣
•
Using Knowledge of Students
to Inform Teaching
Planning Assessments to
Monitor and Support Student
Learning
ENGAGING
‣ Engaging Students in Learning
‣ Deepening Student Learning
During Instruction
ASSESSMENT
‣ Analyzing Student Work
‣ Using Assessment to Inform
Instruction
‣
•
Using Feedback to Guide
Further Learning
REFLECTION
‣ Monitoring Student Progress
and Adjusting Instruction
•
ACADEMIC LANGUAGE
‣ Understanding Language
Demands and Resources
‣
Developing Students’
Academic Language
Repertoire
© Stanford Center for Assessment, Learning, and Equity, 2010
10
Rubric Levels
•
•
Is the candidate ready for independent
teaching
(i.e., to be the teacher of record)?
Rubric Levels
‣ Level 1 – Some skill but needs more
practice to be teacher-of-record
‣ Level 2 – Acceptable level to begin
teaching
‣ Level 3 – Solid foundation of
knowledge and skills
‣ Level 4
– Stellar
candidate,
inLearning,
theandtop
© Stanford
Center for Assessment,
Equity, 2010
Questions and Answers
???
© Stanford Center for Assessment, Learning, and Equity, 2010
12
PACT Scores
Inter-rater Reliability
Level of Agreement
Percent
Exact Match
46%
± 1 point
34%
± 2 points or greater
10%
Sample Size · 2,580
Spearman-Brown Reliability Estimate · 0.88
© Stanford Center for Assessment, Learning, and Equity, 2010
13
PACT Validity Studies
•
‣
‣
•
Factor Analysis
(2002-03 Pilot Year):
• Reflection
& Assessment
TPE alignment study
• Instruction
• Planning
Evaluation of score validity
Decision Consistency ·
Holistic vs. Analytic ratings
Bias and fairness
review
Construct validity
‣
Development teams,
Program directors,
Program faculty, &
Leadership team
Concurrent validity
‣
‣
•
•
Content validity
•
•
Predictive Validity
(Carnegie/CT Study)
Consequential Validity
© Stanford Center for Assessment, Learning, and Equity, 2010
14
Research Plans
•
•
•
Value-added analysis
Observations and interviews of
candidates completing assessment +
first year of teaching
Surveys and interviews with faculty
and program administrators on use of
assessment data
© Stanford Center for Assessment, Learning, and Equity, 2010
15
Assessment Development
•
•
•
•
2009-10 Small-scale tryout tasks & feedback from
users
2010-11 Development of pilot prototypes based on
feedback and piloting in at least 2 credential areas
per institution. User feedback to guide revisions
2011-12 National field test of prototypes, producing
a technical report with reliability and validity studies
and a bias and sensitivity review. National standard
setting.
2012-13 Adoption of validated assessment
© Stanford Center for Assessment, Learning, and Equity, 2010
16
Overview of Tasks and
Rubrics
•
Artifacts and Commentaries
‣ What looks familiar?
•
Level Three Rubric Descriptors
‣ What gets assessed?
•
Tips for implementation
© Stanford Center for Assessment, Learning, and Equity, 2010
17
Handbook Overview
What looks like something we already
do?
What looks new or different?
What will our candidates be good at
now?
What might be a struggle?
© Stanford Center for Assessment, Learning, and Equity, 2010
18
Context Matters
•
Candidates describe what they know about their
students as learners:
‣
‣
‣
‣
‣
•
•
Academic development
Academic Language development
Social-emotional development
Family, community and cultural assets
Special needs
Candidates use information about students to plan
appropriate instruction and assessment.
Scoring rubrics examine extent to which needs of
students are addressed
© Stanford Center for Assessment, Learning, and Equity, 2010
19
Guiding Questions and
Analytic Rubrics
•
PLANNING
‣ Planning Focused, Sequenced
•
Instruction
‣
‣
•
Using Knowledge of Students
to Inform Teaching
Planning Assessments to
Monitor and Support Student
Learning
ENGAGING
‣ Engaging Students in Learning
‣ Deepening Student Learning
During Instruction
ASSESSMENT
‣ Analyzing Student Work
‣ Using Assessment to Inform
Instruction
‣
•
Using Feedback to Guide
Further Learning
REFLECTION
‣ Monitoring Student Progress
and Adjusting Instruction
•
ACADEMIC LANGUAGE
‣ Understanding Language
Demands and Resources
‣
Developing Students’
Academic Language
Repertoire
© Stanford Center for Assessment, Learning, and Equity, 2010
20
EM1: Planning Focused,
Sequenced Instruction
•
•
•
Standards/objectives, learning tasks, and assessments
are clearly aligned to a big idea or essential question.
The learning tasks and assessments represent differing
depths of understanding
Candidate plans how to make clear connections among
mathematical facts, computations/procedures, concepts,
and reasoning.
Learning tasks build on each other to promote an
understanding of the designated mathematical
concepts, computations/procedures, and reasoning
skills. Learning tasks (or their adaptation) are justified
by explaining their appropriateness for students with
references to relevant research and/or theory.
© Stanford Center for Assessment, Learning, and Equity, 2010
21
•
•
EM2: Using Knowledge of
Students to Inform
Teaching
Learning tasks and materials draw upon students’
academic and social/emotional development,
including strengths, as well as experiences and
interests to help students reach the learning
objectives.
Planned support consists of strategically selected
or modified tasks/materials and/or scaffolding of
instruction that is closely tied to specific learning
objectives. It is appropriate for specific individuals
or subgroups.
© Stanford Center for Assessment, Learning, and Equity, 2010
22
EM3: Planning Assessments that
Monitor and Support Student
Learning
•
•
•
The set of assessments will provide evidence of
student learning relative to the standards/objectives
for each lesson. At least one lesson’s assessments
provide evidence of student learning that extends
beyond the formulaic application of computations
or procedures.
Assessments are aligned to clearly defined
benchmarks or criteria for student performance.
Assessments are modified or adapted to be
appropriate for students having difficulty
demonstrating their learning.
© Stanford Center for Assessment, Learning, and Equity, 2010
23
EM4: Engaging Students in
Learning
•
•
Strategies for intellectual engagement seen in the
clip(s) offer structured opportunities for students to
develop their own understanding of mathematical
concepts through discourse. These strategies
reflect attention to students’ academic or language
development, social/emotional development, and/or
cultural and lived experiences.
Candidate identifies successful and unsuccessful
teaching practices. The proposed improvements
are reasonable and address the learning of a
subgroup or individual students.
© Stanford Center for Assessment, Learning, and Equity, 2010
24
EM5: Deepening Student
Learning During Instruction
•
•
Candidates and/or other students build on what
students are saying and/or doing, using reasoning
to improve understanding of mathematical
concepts.
Candidate and/or other students prompt students to
make connections between and among
mathematical concepts and representations of
content.
© Stanford Center for Assessment, Learning, and Equity, 2010
25
EM6: Analyzing Student
Work
•
•
•
Criteria are well-defined and reflect the depth of
understanding stated in of the indicated
standards/objectives from the learning segment.
The analysis focuses on patterns of student errors,
skills, and understandings in relation to standards
and learning objectives. The analysis uses these
patterns to understand student thinking.
The analysis is supported by work samples and the
summary of learning. Specific patterns are
identified for individuals or subgroup(s) in addition
to the whole class.
© Stanford Center for Assessment, Learning, and Equity, 2010
26
EM7: Using Assessment to
Inform Instruction
•
•
•
Next steps follow from an accurate analysis of
student learning and aim at improving student
understanding of important features of the
standards/learning objectives.
Next steps focus on improving student performance
through targeted support to individuals and groups
to address specific identified needs.
Next steps are based on whole class patterns of
performance and some patterns for individuals
and/or subgroups and are described in sufficient
detail to understand them.
© Stanford Center for Assessment, Learning, and Equity, 2010
27
EM8: Using Feedback to
Guide Learning
•
•
Specific and accurate feedback helps the student
understand what s/he did well, and provides
guidance for improvement.
Candidate describes how students will use
feedback to improve their work or their
understanding.
© Stanford Center for Assessment, Learning, and Equity, 2010
28
EM9: Monitoring Student
Progress and Adjusting
Instruction
•
•
Daily reflections identify what students could or
could not do within each lesson and consider the
implications for meeting the standards/objectives at
the end of the learning segment.
Adjustments to instruction are appropriate and
focused on addressing some individual and
collective learning needs.
© Stanford Center for Assessment, Learning, and Equity, 2010
29
EM10: Understanding
Language Demands and
Resources
•
•
•
Candidate describes academic language strengths
and needs of students at different levels of
academic language proficiency.
The language genre(s) discussed are clearly related
to the academic purpose of the learning segment
and language demands are identified. One or more
linguistic features and/or textual resources of the
genre are explicitly identified.
Candidate identifies essential vocabulary for
students to actively engage in specific language
tasks.
© Stanford Center for Assessment, Learning, and Equity, 2010
30
EM11: Developing Students’
Academic Language
Repertoire
•
•
The candidate’s use of scaffolding or other support
provides access to core content while also
providing explicit models, opportunities for
practice, and feedback for students to develop
further language proficiency for selected genres
and key linguistic features.
Candidate articulates why the instructional
strategies chosen are likely to support specific
aspects of students’ language development for
different levels of language proficiency.
© Stanford Center for Assessment, Learning, and Equity, 2010
31
Potentially New
Competencies
•
•
•
•
Academic language development
Interpreting student work and
identifying trends across the class
Designing next steps in instruction
based on assessment results
Providing feedback to students
© Stanford Center for Assessment, Learning, and Equity, 2010
32
Issues to Think about for
Spring 2011 Pilot
•
•
•
•
•
•
•
Match with curriculum
Which faculty? Which credential areas?
Candidate support
Timeline, from introducing assessment
through scoring
Preparation for videotaping
Recruiting scorers
Consideration of results
© Stanford Center for Assessment, Learning, and Equity, 2010
33
What did successful PACT
programs do?
•
•
•
•
•
•
Broad and collegial involvement in decision
making
Inquiry and program improvement rather
than compliance
Affirm and maintain program values and
identity
Broad involvement in assessment activities
Regular practices and protocols for data
analysis
Learned from individual “case” studies
© Stanford Center for Assessment, Learning, and Equity, 2010
34
Questions and Answers
???
© Stanford Center for Assessment, Learning, and Equity, 2010
35
Academic Language
Overview
Three “F” Words…
© Stanford Center for Assessment, Learning, and Equity, 2010
36
Academic Language
Academic language development is making the
language explicit to expand students’ control over
language and improve their language choices
according to the purpose/FUNCTION and audience for
the message.
Academic language also offers structures/FORMS for
developing as well as expressing explanations,
evaluations, and analyses.
Developing students’ FLUENCY in academic language
forms and functions provides access to the “language
of school” and academic success
© Stanford Center for Assessment, Learning, and Equity, 2010
37
Academic Language
The purposes of Academic Language are to
clearly and explicitly define, classify, analyze,
explain, argue, interpret and evaluate ideas
for distant audiences.
© Stanford Center for Assessment, Learning, and Equity, 2010
38
Academic Language
The purposes of Academic Language are to
clearly and explicitly define, classify, analyze,
explain, argue, interpret and evaluate ideas
for distant audiences.
= FUNCTION
© Stanford Center for Assessment, Learning, and Equity, 2010
39
•
•
Academic Language
Language for academic purposes differs greatly from
every day purposes
These differences include
‣
‣
‣
‣
‣
a better-defined system of FORMS (genres) with explicit
expectations about how texts are organized to achieve academic
purposes
precisely-defined vocabulary to express abstract concepts and
complex ideas
more complex grammar in order to pack more information into each
sentence
a greater variety of conjunctions and connective words and phrases
to create coherence among multiple ideas
formatting conventions, graphics and organizational titles and
headings to guide understanding of texts
© Stanford Center for Assessment, Learning, and Equity, 2010
40
Academic Language
Academic Language also includes instructional
language needed to participate in learning and
assessment tasks, including discussing ideas and
asking questions, summarizing instructional and
disciplinary texts, following and giving instructions,
listening to a mini-lesson, explaining thinking aloud,
giving reasons for a point of view, writing essays to
display knowledge on tests.
© Stanford Center for Assessment, Learning, and Equity, 2010
41
Academic Language
in TPAC
•
Vocabulary
•
Genre/forms with particular functions
▪ Linguistic features (grammar,
organization)
▪ Textual resources (headers, table of
contents, illustrations, graphs, charts)
© Stanford Center for Assessment, Learning, and Equity, 2010
42
Vocabulary
•
•
•
Technical vocabulary: triangle, metaphor,
metabolize
Words whose technical meaning is different than
everyday language: “balance” in chemistry,
“plane” in mathematics, “ruler” in history/social
science, “force” in science
Connector words: and, but, because, therefore,
however
© Stanford Center for Assessment, Learning, and Equity, 2010
43
Genres/Forms
•
•
Have a general structure – e.g., narratives,
explanations, arguments
Structure of an explanation
- Description of what is being explained
- Statements of cause-effect relationships
- Sometimes ending with an interpretation
judgment
© Stanford Center for Assessment, Learning, and Equity, 2010
44
Subject-Specific
Genres/FORMS
•
Representing word problems mathematically
•
Procedures for a science experiment
•
Literary interpretation
•
Argument proposing causes of an historical
event
© Stanford Center for Assessment, Learning, and Equity, 2010
45
Structures of
Argument Genre/FORM
•
•
•
Mental verbs used to express opinions: like,
believe
Move from personal to impersonal voice
Connectives used for logical relations and to link
points
‣Temporal connectives: first, next
‣Causal conditional connectives: because
‣Comparative connectives to introduce counterpoints:
consequently, therefore
© Stanford Center for Assessment, Learning, and Equity, 2010
46
Structure of Arguments
(from Knapp & Watkins, 2005)
•
Simple argument: point/proposition, elaboration
‣
•
•
•
“I like The Simpsons because they make me laugh.”
Argument with evidence: Proposition, argument,
conclusion
Discussion: statement of issue, arguments for,
arguments against, recommendation
Elaborated discussion: statement of issue, preview of
pro/con, several iterations of point/elaboration
representing arguments against, several iterations of
point/elaboration representing arguments for, summary,
conclusion
© Stanford Center for Assessment, Learning, and Equity, 2010
47
Characteristics of Advanced Academic Language
•
•
Abstractions: government, electron, linear equation,
acid
Nominalizations, verbs or adjectives becoming
nouns to enable more dense text or more cohesive
text:
organize into…→ this organization…
…were revealed. The trigger for this revelation
was…
•
More precise connector words and phrases, going
beyond “and” or “but” to “in contrast” or “Given
this, it follows that…”.
© Stanford Center for Assessment, Learning, and Equity, 2010
48
Academic Language
Competencies Measured (Rubric 10)
•
Understanding language demands and resources
for instructional emphasis
‣
‣
‣
Identification of linguistic features of a
genre/form/function addressed within instruction
Relation of vocabulary identified to content and to
students’ academic language proficiencies
Description of student language strengths and needs
© Stanford Center for Assessment, Learning, and Equity, 2010
49
Academic Language
Competencies Measured (Rubric 11)
•
Expanding students’ academic language
repertoires
‣
‣
‣
Making key linguistic features related to genre
purpose (form and function) visible to students
Modeling vocabulary and linguistic features
and providing opportunities for practice
(developing fluency)
(at higher levels) appropriateness of models
for students at different levels of language
proficiency
© Stanford Center for Assessment, Learning, and Equity, 2010
50
Activity
•
Analyze video for academic language using graphic
organizer
http://www.learner.org/resources/series33.html
© Stanford Center for Assessment, Learning, and Equity, 2010
51
Academic Language
Takeaways
•
•
•
Academic language is different from everyday
language. Some students are not exposed to this
language outside of school.
Much of academic language is discipline-specific.
Unless we make academic language explicit for
learning, some students will be excluded from
classroom discourse and future opportunities that
depend on having acquired this language.
© Stanford Center for Assessment, Learning, and Equity, 2010
52
Upcoming Webinar
Tuesday, November 16
1:30 Eastern Time
•
To register contact Rachel Popham at
‣ rpopham@aacte.org
© Stanford Center for Assessment, Learning, and Equity, 2010
53