Identifying the Content
Knowledge Needed for
Effective Teaching
Implications for Teacher Preparation,
Accreditation, and Assessment
Heather Howell
Jamie Mikeska
Geoffrey Phelps
Today’s Session
•
•
•
•
Overview of ETS® NOTE
Content Knowledge for Teaching
Group Breakout
Validity Evidence for CKT
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
2
ETS® National Observation
Teaching Exam (NOTE)
• NOTE is an innovative series of assessments
designed to evaluate prospective teachers’
ability to translate their knowledge of content
and teaching into effective practice
• Grounded in research on K-12 teaching
combined with advances in assessment
technology.
• Created in a collaboration between
Educational Testing Service and
TeachingWorks at the University of Michigan
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
3
NOTE Assessments
• content knowledge for teaching (CKT)
assessments: tests of the special ways teachers
need to know content in the core subject areas,
and.
• performance assessments: tests of highleverage teaching practices using “on the spot”
videorecorded performances, including,
• interactive performance assessments: tests
created in collaboration with Mursion to assess
practices that require interaction with a
classroom, with a student, or with a family
member
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
4
NOTE at-a-glance
Elementary
Delivered in test centers
CONTENT
KNOWLEDGE
FOR TEACHING
ASSESSMENTS
PERFORMANCE ASSESSMENTS
“
Modeling and Explaining Content
Delivered using Mursion
simulated classroom with
student and guardian avatars
Interactive Performance Assessments
ELEMENTARY CKT
Mathematics
SUBTEST
Leading Classroom Discussion
ELA
SUBTEST
Science
SUBTEST
Social Studies
Eliciting Student Thinking
SUBTEST
Communicating with a Parent or Guardian
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
5
CONTENT KNOWLEDGE
FOR TEACHING
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
6
What Content Knowledge is
Needed for Effective Teaching?
•
•
•
•
Common content knowledge
Forms of professional content knowledge
Pedagogical Content Knowledge (PCK)
Specialized Content Knowledge (SCK)
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
7
Common Content Knowledge
Alexis has answered the following four
problems in a lesson on order of operations.
Which problems did she answer correctly?
1) 7 x 2 – 6 + 3 = 5
2) 9 – 5 + (16 ÷ 8) = 2
3) 9 + 24 ÷ 3 – 1 = 16
4) 17 – (3 + 7 X 2) = 0
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
8
Specialized Content Knowledge
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
9
Focusing on Content Knowledge
For Teaching
The content knowledge used to recognize,
understand, and respond to the content
problems encountered in teaching
• The work of teaching defines the construct
• By definition a form of applied knowledge
• Includes a wide range of content tasks
and associated content knowledge
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
10
Examples of Mathematical Work of
Teaching
• Selecting, creating, or evaluating
representations to show a particular
mathematical idea
• Selecting or evaluating examples for a
mathematical purpose
• Evaluating whether a particular problem could
support the use of a particular solution strategy
• Determining whether written student work
demonstrates use of a particular mathematical
idea or strategy
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
11
BREAKOUT TO STUDY CKT
TASKS
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
12
Questions for Consideration
• What types of content knowledge are
assessed by the example task?
• Is this content knowledge that matters for
initial licensure?
• What challenges do EPPs face in providing
prospective teachers with opportunities to
develop CKT?
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
13
Secondary Mathematics
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
14
Elementary Science
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
15
Elementary ELA
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
16
Results from Cognitive interviews
VALIDITY EVIDENCE FOR
CKT ASSESSMENTS
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
17
Measures of Effective Teaching
(MET) Cognitive Interview Study
• Design
– 10 CKT items, 30 teachers each subject
– 90 minute interview
– 600 item responses across ELA and math
• Research Questions
– Do correct answers conform to item rationale?
– Are incorrect answers due to difficulty with
content and content teaching?
See Howell, Phelps, Croft, Kirui, Gitomer (2013) for more detail.
Project funded in part by the Bill and Melinda Gates foundation
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
18
Correct Answer
9/15/2015
“If Alexis did it her way she
would have done the three
times two is six, she would
have done the addition first
because in Please Excuse My
Dear Aunt Sally, “Aunt”
comes before “Sally.” So that
would be six plus five is 11
and so then she’d do her
subtraction which is 13 minus
11 and Alexis’ answer would
be two. She would come up
with the incorrect answer
because she would have not
followed the correct order of
operations. Correct in her
mind but not the way it
should be taught.”
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
19
Teaching Experience Supports
Answering Correctly
“Everybody uses that mnemonic Please Excuse
My Dear Aunt Sally to show the order of
operations and that’s what this child is messing
up. She’s adding before subtracting because it
says it in that order. A lot of times I’ve seen
teachers actually teach it wrong where they tell
them you have to go through the addition before
we do subtraction, which of course is incorrect.
You have to do the addition and subtraction
simultaneously from left to right. And that seems
how this child has messed up she’s doing her
addition before she does her subtraction.”
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
20
Incorrect Answer
9/15/2015
Respondent: Because it has
the division in it with the
parenthesis. I figured that if she
was going to make a mistake, it
would be on that one again.
Interviewer: You said that
because there was division with
parenthesis you were looking
for that?
Respondent: Right because
number two was wrong because
of that and the number one
with just the multiplication,
subtraction, addition, I think
that’s a little easier than
number two.
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
21
Teaching Experience Does Not
Support Answering Correctly
“We do order of operations. My kids used the PEMDAS
and there’s a lot of debate about saying the letters or
just saying ‘Please Excuse My Dear Aunt Sally’ which is
the way I know how to get them to remember it. I
don’t know what’s wrong with ‘Please Excuse My Dear
Aunt Sally’ but people don’t want to use it. But it works
all the letters are there parenthesis, exponents,
multiplication, division, addition, subtraction, so I tell
them they should write that down before they even
start doing a problem that has more than one
operation in it so they use the right thing.”
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
22
Comparing Answers to Reasoning
The Santiago Item
Reasoning
conforms to
design rationale
Reasoning
does not
conform
12
2
0
17
Correct Answer
Incorrect Answer
Primary Reasons for Incorrect Answer
9/15/2015
Difficulty with teaching
10
Difficulty with content
6
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
23
Comparing Answers to Reasoning
All elementary Math and ELA Items
N=1180
item-responses
Correct Answer
Incorrect Answer
Reasoning
conforms to
design rationale
Reasoning
does not
conform
734 (62%)
120 (10%)
0
326 (28%)
90% of item responses “on-diagonal”
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
24
Comparing Answers to Reasoning
All elementary Math and ELA Items
N=1180
item-responses
Correct Answer
Reasoning
conforms to
design rationale
Reasoning
does not
conform
734 (62%)
120 (10%)
0
326 (28%)
Incorrect Answer
Primary Reasons for Incorrect Answer
Difficulty with teaching
Difficulty with content
9/15/2015
217 (49%)
134 (30%)
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
25
Comparing Answers to Reasoning
Elementary Science
N=162
item-responses
Correct Answer
Incorrect Answer
Reasoning
conforms to
design rationale
Reasoning
does not
conform
75 (46%)
17 (11%)
8 (5%)
62 (38%)
84% of item responses “on-diagonal”
CKT Science work is supported in part by a grant from the National Science Foundation, DGE-1445641
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
26
Comparing Answers to Reasoning
Secondary Mathematics
N=186
item-responses
Correct Answer
Reasoning
conforms to
design rationale
Reasoning
does not
conform
97 (52%)
20 (11%)
3 (2%)
66 (35%)
Incorrect Answer
90% of item responses “on-diagonal”
Primary Reasons for Incorrect Answer
Difficulty with teaching
Difficulty with content
33 (38%)
40 (47%)
CKT Secondary Mathematics work is supported in part by a grant from the National Science Foundation, DGE-1445630
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
27
Thank You!
For more information on CKT development
Heather Howell (mathematics)
hhowell@ets.org
Jamie Mikeska (science)
jmikeska@ets.org
Geoffrey Phelps (ELA & social studies)
gphelps@ets.org
9/15/2015
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are
registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 31961
28