Mendon-Upton District
District Determined Measures
Dr. Deborah Brady
dbrady3702@msn.com
1. What are
DDMs?
• Communicating
your local
expectations
• Quelling anxiety
• All data stays
in the district
• Evaluation
consequence
s are minimal
2. Assessment
Development
and Quality
•
•
Which tests that
we give now can
we use or adapt?
Defining Quality:
• Alignment
to CCSS
• Rigor
3. K-12 DDMs; 2 DDMs
per educator
4. Implementing,
Collecting, Organizing
and Storing
 What about
“singleton” courses ?
 What about PE, Art,
Music, technology?
 What about SISPs?
 Administrators?
• Who gives the
tests?
• Are the teachers’
grades calibrated?
• Who grades,
stores, and
analyzes the
assessments?
5. Analyzing scores and
determining L, M, H
Impact Scores
•
•
•
•
•
What’s high growth?
What’s a year’s
growth?
What’s low growth?
Where do we store
this data?
How do we use the
results in the
evaluation process?
What are your NEXT STEPS?
5. Analyzing scores and
determining L, M, H
Impact Scores
2. Assessment
Development
and Quality
1. What are
DDMs?
4. Implementing
DDMs, Collecting,
Organizing and Storing
What are your
NEXT STEPS?
3. K-12 DDMs; 2 DDMs
per educator
Agenda
1.
2.
3.
4.
5.
DDM Overview
Defining Quality DDMS
Developing Quality DDMS
Examples of DDMS
Storing and Scoring
• On-line access to materials now at: www.ddmsbrady.wikispaces.com
DDM
Overview
District Determined Measures
DEFINITION
DDMs are defined as:
“Measures of student learning, growth,
and achievement related to the
Curriculum Frameworks, that are
comparable across grade or subject level
district-wide”
TYPES OF MEASURES
• Portfolio assessments
• Approved commercial
assessments
• MCAS ELA 4-8; MCAS Math 4-8
• District developed pre- and postunit and course common
assessments
• Capstone projects
DDM Quick Review
Must measure growth, not achievement
Growth equals one year’s progress
Each educator will have at least 2 DDMs per year and 4 data points (L, M, or H)
Teachers’ DDMs will be assessments of student work called a DIRECT Measure
Most growth will be based on a pre-test before teaching and a post-test after teaching
MCAS SGP for grades 4-8 for ELA and math ONLY can be used (not grade 10)
Scores (100%) or Rubrics can be used to measure progress
One measure must not be MCAS; it must be a District Determined Measure which can include local
assessments, normed assessments (DRA, MAP, SAT). However, self-contained classroom teachers may use
both ELA and math SGA if the district makes that decision.
Some SISPs, administrators, nurses may have to have 1 INDIRECT Measure (a few who do
not work with students may have 2 indirect).
INDIRECT Measures are like SMART goals—attendance, graduation rates, MCAS
Typical Timeline for DDMs and Impact Ratings
Implementation
Last Year
District-wide training, development of assessments and piloting of 5 required DDMs (Primary ELA or math; Elementary ELA or
Math, MS Math, HS Writing to Text plus un-tested area)
June 2014 Report: List of DDMs from District plus any waivers granted by DESE.
2014-2015
Year 1 (1st Cohort) Non-waivered; scores are divided into H-M-and Low and stored locally
Year 2 (2nd Cohort) Areas waivered by DESE based on June report
2015-2016
Second year data is collected for 1st Cohort.
First year of DDM data is collected for the 2nd Cohort.
2016-2017
October 2016: First DDM rating of High Moderate or Low is given to the 1st Cohort. The impact rating linked to the educator’s
EPIMS NUMBER is sent to DESE with the October 1 Report based on 2 years of data for two DDMs.
A second year of data is collected for 2nd Cohort. Their Impact Rating will be calculated and sent to DESE by October 2017.
Performance & Impact Ratings
Performance Rating
Ratings are obtained through data
collected from observations, walkthroughs and artifacts
• Exemplary
• Proficient
• Needs Improvement
• Unsatisfactory
Impact Rating
( October 1, 2016 for 1st Cohort)
Ratings are based on trends and
patterns in student learning, growth
and achievement over a period of at
least 2 years Data gathered from DDM’s
and State-wide testing
• High
• Moderate
• Low
Performance and Impact Ratings
Performance
Rating
Summative Rating
Exemplary
1-yr Self-Directed
Growth Plan
2-yr Self-Directed Growth Plan
Proficient
Needs
Improvement
Directed Growth Plan
Unsatisfactory
Improvement Plan
Low
Moderate
High
Rating of Impact on Student Learning
Impact
Massachusetts Department of Elementary and Secondary
Education
Rating
10
Defining Quality Assessments
Assessments
• What is needed?
• 2 per educator per year
• Chart of assessments and general descriptions
• Quality Assessment Development
• Collection and Approval (Form)
– Criteria
– Storing, organizing
• Pre and Post testing
– Collection of data
– Analysis of results
– Low, Moderate and High
Quality Assessments
• “Substantive”
– Aligned with at least 2 standards of Frameworks
– And/or local standards
•
•
•
•
Rigorous
Consistent in substance, alignment, and rigor
Consistent with the District’s values, initiatives, expectations
Measure growth (to be contrasted with achievement) and shifts the focus
of teaching toward the individual learner’s change over time
Quality Assessments
Almost any kind of assessment can work
Must be a “Substantive” assessment (DESE)
Aligned with (at least 2) standards of Frameworks or 1 Power Standard
And/or local standards
Rigorous (appropriately challenging; locally defined)
Consistent with K-12 DDMs in substance, alignment, and rigor
Consistent with the District’s values, initiatives, expectations
Measures growth (to be contrasted with achievement)
How will you collect and organize DDMs?
Sample Plan Part I
SGP and Local Common Assessments*
ELA
Math
Science
Social Studies
12
CA/CA
CA/CA
CA/CA
CA/CA
11
CA/CA
CA/CA
CA/CA
CA/CA
10
CA/CA
CA/CA
CA/CA
CA/CA
9
CA/CA
CA/CA
CA/CA
CA/CA
8
MCAS SGP/CA
MCAS SGP/CA
CA/CA
CA/CA
7
MCAS SGP/CA
MCAS SGP/CA
CA/CA
CA/CA
6
MCAS SGP/CA
MCAS SGP/CA
CA/CA
CA/CA
5
MCAS SGP/CA
MCAS SGP/CA
4
MCAS SGP/CA
MCAS SGP/CA
3
CA/CA
CA/CA
2
CA/CA
CA/CA
1
CA/CA
CA/CA
K
CA
CA
Common Assessments
or Locally Determined
Singleton, Art, Music,
Technology, PE
by Grade or grade-spans
Special Education
Specialists, Co-Teachers,
substantially separate
Indirect Measures (IM)
Central Office,
Psychologist
Administrators
Principals, Assistant
Principals,
Superintendent,
Coordinators
Inclusion Co-Teachers
Can “share” scores with
General Ed Teachers
Adaptation of the SMART
goal process
MCAS SGP
Either ELA or Math
4-10
12
CA/CA
11
CA/CA
10
CA/CA
9
CA/CA
8
CA/CA
Or
7
CA/CA
6
CA/CA
5
CA/CA
4
CA/CA
If their students’ goals are
substantially different, the
assessments can be
modified or can focus on
the goal of inclusion
3
CA/CA
2
CA/CA
1
CA/CA
K
CA/CA
Specialists Measure Goals
Can develop K-12 rubric
Speech
PE
Measure goal that has an
impact on student growth
Attendance
College Applications
Technology Growth
Plus other direct
measures
PLUS Indirect Measures
Planning and Organizing
Handouts
• DDM Educator Allocation by School or Department
• DDM Cover Sheet for Each DDM
• DDM Quality Checklist
• MURSD Assessment Checklist…how to connect the work you are
already doing?
Other
• Development of Data Teams
• Self-assessment of assessment literacy skills
Table Talk on Planning
Developing Quality Assessments
Five Requirements of DDMs (DESE)
1. Must measure growth
2. Employ a common administration procedure
3. Use a common scoring process
4. Translate these assessments to an Impact Rating (High-Moderate-Low)
5. Assure comparability of assessments within the school (alignment, rigor,
validity).
20
Measures Growth:Student Growth Percentiles
(SGP Calculation for individual students)
288 to 244/ 25 SGP
4503699
230 to 230/ 35 SGP
214 to 225/ 92 SGP
Measures Growth: MCAS SGP
ELA/Math for Grades 4-8 (not 10)
Small Classes, Co-Taught Classes
• The median SGP must be used when a teacher has 20 or more students
(altogether) in a content area (ELA or mathematics 3-8)
• Median SGPs for 8-19 students have validity and may be used if the
district determines this is appropriate
• More than one educator (a classroom teacher, a support teacher, a
curriculum supervisor, and a supervising administrator) may be considered
responsible for a content-area SGP. Different students may be in their
respective rosters.
Measures Growth: “Cut Scores” for MCAS SGP
Typical
growth
One year’s growth
Higher
growth
Lower
growth
0
50
100
65
35
Classroom
40
Whole Grade
Percent of students
60
Whole Grade
Classroom
Rigor
• Bloom (next slide)
• Hess (next slide)
• DESE’s “Curriculum Embedded Performance Assessment”
Rubric in the Model Curriculum Units
DOK Hess: Rigor Relevance Matrix Cognitive
Level 3 Complexity
Depth +
thinking
Level 1
Recall & Reproduction
Remember
-Recall, locate basic facts,
details, events
Understand
-Select appropriate words to use
when intended meaning is clearly
evident
B
Analyze
M
Not appropriate at this level
-Specify or explain relationships
-summarize
-identify central idea
-Explain, generalize, or connect ideas
using supporting evidence (quote,
example…)
-Explain how concepts or ideas
specifically relate to other
content domains or concepts
-Use context to identify meaning
of word
-Obtain and interpret information
using text features
-Use concepts to solve non-routine
problems
-Devise an approach among
many alternatives to research a
novel problem
-Compare literary elements,
terms, facts, events
-analyze format, organization, &
text structures
-Analyze or interpret author’s craft
(literary devices, viewpoint, or
potential bias) to critique a text
-Analyze multiple sources
-Analyze complex/abstract
themes
-Cite evidence and develop a logical -Evaluate relevancy, accuracy,
& completeness of information
argument for conjectures
PARCC
-Brainstorm ideas about a topic
Create
Strategic Thinking/
Reasoning
MCAS
-Use language structure
L (pre/suffix) or word relationships
Apply
(synonym/antonym) to determine
O meaning
-Identify whether information is
O contained in a graph, table, etc.
Evaluate
Level 2
Skills & Concepts
Level 4
Extended Thinking
-Generate conjectures based on
observations or prior knowledge
-Synthesize information within one
source or text
-Synthesize information across
multiple sources or texts
GENERIC Rubric for CEPAs in Mass
1
Topic
development
2
Little topic/idea
development,
organization, and/or
details
Little or no awareness of
audience and/or task
Evidence and
Content Accuracy
Little or no evidence is
included
and/or
3
4
5
Limited or weak topic/idea Rudimentary topic/idea Moderate topic/idea
development, organization, development and/or
development and
and/or details
organization
organization
Limited awareness of
audience and/or task
Basic supporting details
Adequate, relevant details
Simplistic language
Some variety in language
Use of evidence and content Use of evidence and
Use of evidence and
knowledge is limited or
content is included but is accurate content is
weak
basic and simplistic
relevant and adequate
Full topic/idea
development
Rich topic/idea
development
Logical organization
Careful and/or subtle
organization
Strong details
Appropriate use of
language
Use of evidence and
accurate content is
logical and appropriate
content is inaccurate
Use of
Visuals/Media
Visuals and/or media are
missing or do not
contribute to the quality
of the submission
Visuals and/or media
demonstrate a limited
connection to the
submission
1
Standards for
English
Conventions
Errors seriously interfere with
communication
and
Little control of sentence structure,
grammar and usage, and mechanics
Visuals and/or media are
basically connected to
the submission and
contribute to its quality
Visuals and/or media are
connected to the
submission and contribute
to its quality
6
Visuals and/or media
contribute to the quality
of the submission in a
logical and appropriate
way
Effective/rich use of
language
A sophisticated selection
of and inclusion of
evidence and accurate
content contribute to an
outstanding submission
Visuals and/or media are
carefully and strategically
selected to enhance the
content of the submission
2
3
4
Errors interfere somewhat with
communication
and/or
Too many errors relative to the length
of the submission or complexity of
sentence structure, grammar and usage,
and mechanics
Errors do not interfere with
communication
and/or
Few errors relative to length of
submission or complexity of sentence
structure, grammar and usage, and
mechanics
Control of sentence structure, grammar
and usage, and mechanics (length and
complexity of submission provide
opportunity for student to show control
of standard English conventions)
Common Administration and Scoring:
Calibrating
• Equal Opportunity for All students: Floor and Ceiling Effects
• Scoring Guides and Directions
• Calibrating Scorers with Rubrics
Rubric Cautions: Holistic Rubric
Show Progress across a Scale, Continuum, Descriptors
1
3
4
No improvement in the level of
detail.
Modest improvement in the level of
detail
Considerable Improvement in the
level of detail
Outstanding Improvement in the
level of detail
One is true
One is true
All are true
All are true
* No new details across versions
* There are a few details included
across all versions
* There are many examples of added
details across all versions,
* On average there are multiple details
added across every version
* There are many added details are
included, but they are not included
consistently, or none are improved or
elaborated upon.
* At least one example of a detail that
is improved or elaborated in future
versions
* There are multiple examples of
details that build and elaborate on
previous versions
*Details are consistently included in
future versions
* The added details reflect the most
relevant and meaningful additions
* New details are added, but not
included in future versions.
* A few new details are added that are
not relevant, accurate or meaningful
Details
2
* There are many added details, but
several are not relevant, accurate or
meaningful
*The added details reflect relevant and
meaningful additions
Example taken from Austin, a first grader from Answer Charter School
in Boise, Idaho. Used with permission from Expeditionary Learning.
Learn more about this and other examples at
http://elschools.org/student-work/butterfly-drafts
28
2
2
2
Preconventional
Emerging
Developing
Beginning
Expanding
Ages 3-5
Ages 4-6
Ages 5-7
Ages 6-8
Ages 7-9
Relies primarily on pictures to
convey meaning.
Begins to label and add “words”
to pictures.
Writes first name.
Uses pictures and print to convey
meaning.
Writes words to describe or support
pictures.
Copies signs, labels, names, and
words (environmental print).
Writes 1-2 sentences about a topic.
Writes names and familiar words.
.
2
2
2
Writes several sentences about a
topic.
Writes about observations and
experiences.
Writes short nonfiction pieces
(simple facts about a topic) with
guidance.
Bridging
Fluent
Proficient
Connecting
Ages 8-10
Ages 9-11
Ages 10-13
Ages 11-14
2
Writes about feelings and opinions.
Writes fiction with clear beginning,
middle, and end.
Writes poetry using carefully chosen
language with guidance.
2
Writes organized nonfiction pieces
(e.g., reports, letters, and lists) with
guidance.
2
Begins to use paragraphs to organize
ideas.
2
Uses strong verbs, interesting
language, and dialogue with
1
guidance.
1
1
Begins to write organized fiction
and nonfiction (e.g., reports,
letters, biographies, and
autobiographies).
Develops stories with plots that
include problems and solutions
with guidance.
Creates characters in stories with
guidance.
Writes poetry using carefully
chosen language.
Begins to experiment with
sentence length and complex
sentence structure.
Varies leads and endings with
guidance.
Uses description, details, and
similes with guidance. 1 Uses
dialogue with guidance.
Writes persuasively about ideas,
feelings, and opinions.
Creates plots with problems and
solutions.
Begins to develop the main characters
and describe detailed settings.
Begins to write organized and fluent
nonfiction, including simple
bibliographies.
Writes cohesive paragraphs including
reasons and examples with
guidance.
Uses transitional sentences to connect
paragraphs.
Varies sentence structure, leads, and
endings.
Begins to use descriptive language,
details, and similes.
Uses voice to evoke emotional
response from readers.
Begins to integrate information on a
topic from a variety of sources.
2
2
2
2
1
1
1
1
Writes in a variety of genres and
forms for different audiences and
purposes independently.
Writes short fiction and poetry with
guidance.
Writes a variety of short nonfiction
pieces (e.g., facts about a topic,
letters, lists) with guidance.
Writes with a central idea.
Writes using complete sentences.
Independent
2
Writes organized, fluent, accurate,
and in-depth nonfiction, including
references with correct
bibliographic format.
2
Writes cohesive, fluent, and
effective poetry and fiction.
Uses a clear sequence of
paragraphs with effective
transitions.
Begins to incorporate literary
devices (e.g., imagery, metaphors,
personification, and
foreshadowing).
Weaves dialogue effectively into
stories.
Develops plots, characters, setting,
and mood (literary elements)
effectively.
Begins to develop personal voice
and style of writing.
.
Creates plots with a climax.
1
Creates detailed, believable
settings and characters in stories.
Writes organized, fluent, and
1
detailed nonfiction independently,
including bibliographies with
correct format.
Writes cohesive paragraphs
1
including supportive reasons and
examples.
1
Uses descriptive language, details,
similes, and imagery to enhance
ideas independently.
1
Begins to use dialogue to enhance
character development.
"
Incorporates personal voice in
writing with increasing frequency. ?
J
AP Rubric of Rubrics Prose Analysis
(9 levels give students room to improve Holistic)
9-8 Answers all parts of the question completely. Using specific evidence from the work and showing how that evidence is relevant to the
point being made. Fashions a convincing thesis and guides reader through the intricacies of argument with sophisticated transitions.
Demonstrates clear understanding of the work and recognizes complexities of attitude/tone. Demonstrates stylistic maturity by an effective
command of sentence structure, diction, and organization. Need not be without flaws, but must reveal an ability to choose from and control a
wide range of the elements of effective writing.
7-6 Also accurately answers all parts of the question, but does so less fully or effectively than essays in the top range. Fashions a sound thesis.
Discussion will be less thorough and less specific, not so responsive to the rich suggestiveness of the passage or precise in discussing its impact.
Well written in an appropriate style, but with less maturity than the top papers. Some lapses in diction or syntax may appear, but demonstrates
sufficient control over the elements of composition to present the writer’s ideas clearly. Confirms the writer’s ability to read literary texts with
comprehension and to write with organization and control.
5 Discusses the question, but may be simplistic or imprecise. Constructs a reasonable if reductive thesis. May attempt to discuss techniques
or evidence in the passage, but may be overly general or vague. Adequately written, but may demonstrate inconsistent control over the
elements of composition. Organization is attempted, but may not be fully realized or particularly effective.
4-3 Attempts to answer the question, but does so either inaccurately or without the support of specific evidence. May confuse the attitude /
tone of the passage or may overlook tone shift(s) or otherwise misrepresent the passage. Discussion of illustrations / techniques / necessary
parts of the prompt may be omitted or inaccurate. Writing may convey the writer’s ideas, but reveals weak control over diction, syntax, or
organization. May contain many spelling or grammatical errors. Essays scored three are even less able and may not refer to illustrations /
techniques at all.
2-1 Fails to respond adequately to the question. May misunderstand the question or the passage. May fail to discuss techniques / evidence
used or otherwise fail to respond adequately to the question. Unacceptably brief or poorly written on several counts. Writing reveals consistent
weakness in grammar or other basic elements of composition. Although may make some attempt to answer the question, response has little
clarity and only slight, if any, evidence in its support. Although the writer may have made some attempt to answer the prompt, the views
presented have little clarity or coherence; significant problems with reading comprehension seem evident. Essays that are especially inexact,
vacuous, and /or mechanically unsound should be scored 1.
0
A blank paper or one that makes no attempt to deal with the question receives no credit.
Rubric from Sharon Kingston
Criterion Referenced Rubric and Raw Scores or % of 100
4(25)=
100
x
4(22)=
88
x
4(18)=
72
4(15)=
25
60
x
+
18
+
22
+
x
15 = 80%
Comparable Assessments
Comparable across schools and within grades
– Example: Teachers with the same job (e.g., all 5th grade teachers or all teachers of
English 8) give the same assessment
– Where possible, measures are identical
• Easier to compare identical measures
• But the district can determine whether or not these identical measures provide meaningful
information about all students
– Exceptions: When might assessments not be identical?
• Different content (different sections of Algebra I)
• Differences in untested skills (reading and writing on math test for ELL students)
• Other accommodations (fewer questions to students who need more time)
• NOTE: MCAS modifications and accommodations based on IEPs and 504s are appropriate
32
Comparability:
Co-Taught Classes and Teacher of Record
• In some co-taught classes or when teachers share in the
results of the assessment because of additional time spent in
a pull-out class, districts may need to determine the teacher
or teachers responsible for content covered by statewide
testing
• Co-teachers may share equally if both co-teach all students
or the district may determine one teacher as the teacher of
record.
Comparability: Roster Verification
When DDM Results Have Been Calculated
Educators confirm the accuracy of their rosters
• Student must be on roster by October 1, and the student must remain
on roster through last day of DDM testing.
• Student must be present for 90% of instructional time.
– DESE recommendation in Model Contract Language
Direct Measures
Direct measures of student learning, growth, or
achievement provide information about student growth
from student work.
Most classroom teachers will have 2 direct measures
More Examples
CCSS Math Practices: A HS math department’s use of PARCC
examples that require writing asking students to “justify your
answer”
SS Focus on DBQs and/or PARCC-like writing to Text: A social
studies created PARCC exam using as the primary sources.
Another social studies department used “mini-DBQs” in
freshman and sophomore courses
Common Criteria Rubrics for Grade Spans: Art (color, design,
mastery of medium), Speech (developmental levels)
Some GOOD Common Exam Examples
• A Valued Process: PORTFOLIO: 9-12 ELA portfolio measured by a
locally developed rubric that assesses progress throughout the four
years of high school
• K-12 Writing or Writing to Text: A district that required that at
least one DDM was “writing to text” based on CCSS appropriate
text complexity
• Focus on Data that is Important: A HS science department
assessment of lab report conclusions for each course (focus on
conclusions)
Indirect Measures for Guidance, Some Caseload
Educators, Some Administrators
 Indirect measures of student learning, growth, or achievement
provide information about students from means other than
student work.
 These measures may include student record information (e.g.,
grades, attendance or tardiness records, or other data related to
student growth or achievement such as high school graduation or
college enrollment rates).
 ESE recommends that at least one of the measures used to
determine each educator’s student impact rating be a direct
measure and MCAS SGP if available and appropriate.
The Process for Developing Indirect Measures
• Consider the teams that you are a part of, for example, what many schools call the “child study team” in
which many specialists participate, or all of your job-alike colleagues.
• Discuss the goals that you have as a district group or as a school-based group. For example, you may be
working on increasing the attendance of a group of students who are frequently out or tardy. Or, you may be
working to return students to class quickly. Or, you may be focusing on working with a few families or
students to support students with, for example, school anxiety.
• Note that the measures can focus on a small group and do not have to include all students. For example,
students with emotional problems that prevent them from participating fully can be your subgroup. For
those educators with a small caseload, supporting your caseload to return to class and to participate fully in
the class may be your goal.
• Select a goal on something that is already an initiative or is recognized as something worthwhile for students.
This goal needs to connect indirectly to student growth. Making sure students are in class and are able to
participate fully is an appropriate goal.
• Social-emotional growth is an appropriate direct measure.
• Develop a method for measuring this goal. For example, nurses in one district realized that the software that
they used could also track their goal of quick returns of students to classes. In another district, a schoolbased team decided that their “Student Support Team” had been reviewing numbers of classroom incidents
and absences, but had not put aside time to study the most at risk students fully. Their goals was to change
their weekly process so that time was set aside to discuss a student more thoroughly and to develop an
action plan for that student. Their record was their weekly agenda, and they counted the increased number
of action plans and tracked the progress of the most at risk students.
• See DESE’s Short Brief on SISP DDMs
Indirect Measures More Specifics:
• Librarians: Increase the number of classes that work on research and projects in the library. Increase
the number of teachers with whom you work to support specific units with materials.
• Behavior specialists, speech pathologists, social workers, school psychologists: Increase the number
of students who participate in class more fully. You can look at a sub-set of your students or caseload.
For example, a behavior specialist was going to count the number of responses to a “non-classroom
adult” that students made as they walked to gym or to lunch. This is a DIRECT MEASURE.
• A group of speech pathologists used a developmental scale (like a holistic rubric) to measure the
improvement of students’ speaking skills. They assessed these skills in their classroom and in general
education classrooms. This is a DIRECT MEASURE.
• If IEP goals measure student academic or social growth, attainment of success in working toward these
goals can be used as DIRECT MEASURES. Thus, achieving IEP growth goals for my caseload can be a
goal. Please note that DESE is still looking at subgroup sizes under 7 as being too small for growth
measurement.
• Guidance counselors set increasing the number of applications for college as a goal. Their
comprehensive initiative included visiting classrooms and encouraging more students to take the PSAT,
to work with the administration to provide SAT workshops, and to offer workshops through the school
in writing college application letters.
Indirect Measures
– High school SST team example (Frequent Absentees)
– Child Study Team example (Universal Process)
– School Psychologists (Did not follow procedure for referral)
– School Psychologists (subgroup of students studied)
– High school guidance example (PSAT, SAT, College Applications)
– IEP goals can be used as long as they are measuring growth
(academic or social-emotional)
Examples of DDMs
The GOOD
The BAD
and the Ugly
Quality Assessments, Developed Locally, Adapted, or Adopted
Dr. Deborah Brady
dbrady3702@msn.com
History-Social Studies
Read Like an Historian
Table Vote
Thumbs UP? Halfway? DOWN
Morale after Fredericksburg
This assessment asks students to source and corroborate a letter from a Union soldier
describing low morale among Union soldiers after the Battle of Fredericksburg. Question
1 asks students to evaluate whether the source provides sufficient evidence to
demonstrate the morale of the entire Union Army. To answer this question, students
must source the document to determine whether the author represents all Union
troops. Question 2 asks students to evaluate whether additional documents would
corroborate the letter. https://beyondthebubble.stanford.edu/assessments
http://sheg.stanford.edu/rlh Stanford University Reading History Like a Historian
H-SS: Mini Q: What Caused the Dust
Bowl?
Overview: In the 1930s America was hit by very bad times. These
were the years of the Great Depression. In cities and in small towns
across the land, banks were failing, businesses were closing, and
workers were being fired. But in some ways farmers were hit hardest
of all, and few farmers were hit harder than those in the Southern
Great Plains. This region has come to be known as the Dust Bowl.
The question for this Mini-Q is what caused the Dust Bowl tragedy.
The Documents:
Document A: Dusters
Document B: Grass
Document C: Fred Folkers and his tractor (with photograph)
Document D: Acreage Under Plow (chart)
Document E: Rainfall on the Plains in the 1930s
Table Vote
Thumbs UP? Halfway? DOWN
Demonstrating Growth (Galileo Example)
Billy Bob’s work is shown below. He has made a mistake In the space to the right,
solve the problem on your own on the right. Then find Billy Bob’s mistake, circle it
and explain how to fix it.
Billy Bob’s work
½ X -10 = -2.5
+10 = +10
Your work
Thumbs UP? Halfway?
DOWN
Find the mistake provides
students with model.
Requires understanding.
Requires writing in math
_____________________________________________
½ X +0 = +12.5
(2/1)(1/2)X =12.5 (2)
X=25
Table Vote
Explain the changes that should be
made in Billy Bob’s Work
Table Vote
Science Lab Rubric—One Focus
Thumbs UP? Halfway? DOWN
Advanced
• Assess student
growth in
writing
conclusions
for labs in all
of the sciences
Conclusions
1. Summarizes
data used to draw
conclusions
2. Conclusions
follow data (not
wild guesses or
leaps of logic),
3. Discusses
applications or
real world
connections
4. Hypothesis is
rejected or
accepted based
on the data.
Proficient
3 of 4 of the
"excellent"
conditions is
met
NI
Warning
2 of the 4
1 of the 4
excellent
excellent
conditions met conditions met
Learning Skills Criteria (Special Education)
• Individual Goals; measured weekly;
permanent folder
–
–
–
–
–
–
–
–
–
–
Notes
Planner
Work/Action Plan
Flexible when Necessary
Prepared for Class (materials, work)
Revises work
Follows instructions
Uses time well
Gets to work
Asks for help when needed
– Advocates for self
– Moving toward independence
– Works collaboratively
Table Vote
Thumbs UP? Halfway? DOWN
Table Vote
Thumbs UP? Halfway? DOWN?
Are numbers
good or a
problem?
Scoring Guides from Text
A scoring guide from a
textbook for building a
Lou Vee Air Car. Is it good
enough to ensure interrater reliability?
• Lou Vee Air Car built to specs (50
points)
• Propeller Spins Freely (60 points)
• Distance car travels
–
–
–
–
1m 70
2m 80
3m 90
4m 100
• Best distance (10,8,5)
• Best car(10,8,5)
• Best all time distance all classes (+5)
Table Vote
Thumbs UP? Halfway? DOWN?
• 235 points total
Example: Generic SISP Rubric for Team Process
Indirect
Criteria
Improving assessment
process for pK-2 referrals
for Special Education
(for quality/process improvement)
Present
Low Improvement
Many students are referred to be
tested for SPED without first
providing classroom-based
interventions.
The process varies among all of
the primary schools
Moderate Improvement
(what is hoped for)
High Improvement
(more improvement than
expected)
All Student Support Teams
All referrals result only after
will use a consistent process Level I interventions have
been tried and assessed for
The specialists and SPED
at least 6 weeks.
staff will provide consulting
or workshops support the
classroom RTI process
Table Vote
Thumbs UP? Halfway? DOWN
PE Rubric in
Progress.
Grade 2 for
overhand
throw and
catching.
Table Vote
Thumbs UP? Halfway? DOWN?
Music: Teacher and Student Instructions
Table Vote
Thumbs UP? Halfway? DOWN?
The Ugly
Comply with regulations
Bring about no change or understanding
The Best
• Build on what is in the District, school or
department
• A small step or a larger step in cognitive
complexity
• Use the results to learn about students’
needs and how to address these needs
• Use time to look at student work, to
collaboratively plan to improve
Scoring and Storing
Scoring and DDM Design Considerations
• Districts will need to determine fair, efficient and accurate methods for scoring
students’ work.
• DDMs can be scored by the educators themselves, groups of teachers within the
district, external raters, or commercial vendors.
• For districts concerned about the quality of scoring when educators score their own
student’s work, processes such as randomly re-scoring a selection of student work to
ensure proper calibration or using teams of educators to score together, can improve
the quality of the results.
• When an educator plays a large role in scoring his/her own work, a supervisor may also
choose to include the scoring process into making a determination of a Student
Impact.
• NOTE: Teacher Instructions may be necessary
Sample Cut Score Determination (for 100 students)
Student Scores
Pre-test
Post test
Difference
20
35
15
5
25
30
5
15
30
50
20
20
35
60
25
25
35
60
25
40
70
40
Sorted low to
high
Teacher score is based on the MEDIAN
Score of her class for each DDM
Cut score
LOW Growth
Lowest ___%
25
median
teacher score
35
25
median
Teacher score
65
25
25
50
75
25
30
50
80
30
35
50
85
35
35
Top 20%
Cut score
HIGH GROWTH
Highest ___?
Measuring Growth Example: Fountas and Pinnell
P to Q; N to P; D to K
Fountas and Pinnell
Growth for Each Student
Is Based on 10 Months of Growth
Second Grade
Student
Level
Achievement
Level End of Year
Levels from
beginning to the end
of the year
Pre-Post F&P Levels
Growth
HIGH, MODERATE, OR
LOW GROWTH
(10 MONTHS=YEAR)
Q
Above Benchmark
PQ
7 MONTHS GROWTH
LOW GROWTH
P
At Benchmark
NOP
10 MONTHS OF
GROWTH
MODERATE GROWTH
K
Below Benchmark
DEFGHIJK
17 MONTHS OF
GROWTH
HIGH GROWTH
Teacher A
Teacher B
Teacher C
Teacher D
Teacher E
Teacher F
103 Third Graders
All Classes
1
3
3
3
5
7.5
6.5
3.5
9
3.5
5.5
6
6.5
6.5
5.5
6.5
6.5
6.5
6.5
7
10
12
1
3
3
6
6.5
7.5
8.5
10
10
10
10
10
6.5
7.5
8.5
9
10
10
10
7
7
7
7
10
10
11.5
10
10
10
10
12
12.5
12.5
6.5
10
10
12.5
12
6.5
6.5
6.5
6.5
6.5
6.5
6.5
6.5 Cut Score lowest 15%
7
7
7
7
7
7
7
7
7.5
7.5
8.5
8.5
9
9
9
9
9
9.5
10
10
10
None
7
7
12
12
10
12
12
16
13
13
9
9
10
10
12
12
12
12
13
13
13
16
17
17
12
13.5
13.5
13.5
17
16
16
16
16
16.5
16.5
16.5
13.5
13.5
13.5
13.6
15.5
19
10
10
10
10.7
10.9
11.0
11.2
11.4
11.5
11.7
11.9
12.0
12.2
12.5
12.6
12.7
12.9
13.1
13.2
13.4
12.0 Median for whole Grade
3 DDM
13.6
13.8
13.9
14.1
14.3
14.4
14.6
14.8
15.0
15.1
15.3
12
16
Median 6.5
Median 9
15.5
17
17
Median 12
Median 10
Median 16
15.6
12.5
12.5
12.5
13
13
Median 12
13
13
13
13.5
13.5
13.5
13.5
Below 6.5
Between 6.5 Between 6.5 Between 6.5
and 16
and 16
and 16
Between 6.5
and 16
Between 6.5
and 16
13.5
13.5
13.6
15.5
16
16
16
16
16
16
16
16.5
16.5
16.5
LOW
Moderate
Moderate
Moderate
High
Moderate
16.5
16.5
17
17
17
19
26
16 cut score highest 15%
Most Impact Ratings are M’s (K Writing)
Teacher A Teacher B Teacher C Teacher D Teacher E
2
0
2
1
1
2
2
2
2
2
3
2
3
2
2
3
2
3
2
2
3
2
3
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 Median
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
3
3
3
3
4
3
3
3
3
4
3
3
4
4
4
4
4
4
4
4
4
4
Post Test Only
Use District’s History to predict scores on AP Calculus Exam, for
example
Previous Grade’s
Math Score
Low
Growth
Moderate
Growth
High
Growth
A
3
4
5
B
2
3
4
C
1
2
3
1
2
D
Considerations for Scoring Student Work
• Districts will need to determine fair, efficient and accurate methods for scoring
students’ work. (Use consistent directions for teachers.)
• DDMs can be scored by the educators themselves, groups of teachers within the
district, external raters, or commercial vendors.
• For districts concerned about the quality of scoring when educators score their own
student’s work, processes such as randomly re-scoring a selection of student work to
ensure proper calibration or using teams of educators to score together, can improve
the quality of the results.
• When an educator plays a large role in scoring his/her own work, a supervisor may also
choose to include the scoring process into making a determination of a Student
Impact.
Summary DDM Process for Determining L, M, H
for Every Teacher
Whole Grade Level or Course
• Score the entire grade level or course or take the MCAS Growth Scores for all students
• Rank the scores from highest to lowest (post minus pre or MCAS SGP)
• Identify the median score for the entire group
• Determine the “cut” scores for local assessments; MCAS 35 and 65 for classrooms
Individual Teacher
• Select students for each teacher
• Rank the scores from highest to lowest
• Identify the Median score
• Is the median below or above the “cut” score? Is it in the middle?
• Don’t forget Roster Verification might change the specific scores and, therefore, change the Median
• Distribute scores to teachers for each DDM
Next Steps in Storing
• Gather data/tests themselves for analysis
– MCAS SGP on line
– Local Assessments
– Transfer to Excel
• Students
• Pre and Post or MCAS SGP or Post test only
• Cut scores
• Determine whole group’s median
• Determine teacher’s median
• Roster Verification
• Determine Low, Medium, or High Growth score for each teacher for each DDM