Combining Multiple Measures
•
•
•
•
What are the indicators/ components?
What are the priority outcomes?
What are the performance expectations?
How can we evaluate effectiveness?
1
Components
• Indicators are specific measures in the system
such as performance on assessments or
graduation rate.
• Components refer to broader (non-mutually
exclusive) categories and can include:
–
–
–
–
–
Achievement (Status)
Growth
Equity
Readiness
Inclusion
2
What to Report
• Classifications
– How many levels?
• How will you define these levels?
• ‘Monikers’ rarely have intuitive meaning. Consider
starting with policy descriptors for each classification
– Scores
• Report outcomes at indicator, component, or
overall level?
3
How to combine?
• There are many approaches to combining within and across
components.
• The ‘how’ is much easier than the ‘why’
– What should outcomes look like and how does this fit into
the overall theory of action?
• Strike balance between simple – flexible
• Methods for combining multiple parts into a whole that yields
a single outcome include:
– Conjunctive
– Disjunctive
– Compensatory
– Profile
4
Conjunctive
• A conjunctive rule means that ALL parts have to
meet a standard in order to achieve a target
outcome.
• Conjunctive rules are used when each part is
distinct and necessary for the outcome.
• Example:
– Currently, NCLB is a conjunctive model in that each
subgroup must meet target performance in each
area for a school to make AYP.
5
Disjunctive
• A disjunctive rule means that ANY part can meet an
established standard in order to achieve an overall
target outcome.
• A disjunctive rule should be used when each
component represents a similar or equally valued part
of the whole. In other words, there are multiple ways
to show quality and any one way is as good as another.
• Example:
– If school meets the target in status OR growth the
overall standard is achieved.
6
Combine Conjunctive/ Disjunctive
• Blend elements of each into overall decision
• For example:
– To be classified as meeting standards…
1. schools should meet EITHER growth or
proficiency standards (disjunctive)
2. these standards have to be met for the whole
school AND all subgroups (conjunctive)
7
Compensatory
• A compensatory approach means that each component
contributes to the outcome, but being higher on one
component can offset or ‘compensate’ for being lower on
another.
• Example:
– Take the average of each component and base the
outcome on the resulting value.
– Produce an index, which can be weighted to reflect
priorities (e.g. 70% growth, 30% status)
– Can set a ‘floor’ or conditions to prevent unintended
outcomes (e.g. average must equal X unless Y is less
than…”)
8
Profile
• Yet another way to combine indicators/ components
is to create profile descriptions.
• A profile refers to a pattern of performance
determined to meet the target or not.
• This approach is particularly useful when
combinations of points are ‘conditional’.
• For example, a combination of 1,1,2 = 1; while a
combination of 1,2,1 = 2.
• In an index, these combinations might yield the same
points, which may be undesirable, because HOW the
points are earned matters.
9
Equity and Excellence
• Who are the students for whom equity concerns are greatest?
– Demographic factors
• Multiple subgroups
• ‘Super-subgroup’
– Performance factors
• Normative (e.g. all students in lowest 25%)
• Criterion (e.g. all students below proficient)
•
– What outcomes should be prioritized
• Attainment of status
• Growth rate
Principles:
– Schools that are already successful – those that have no or very small
achievement gaps – should not be disadvantaged.
– Schools that have the largest achievement gaps should have the most incentive
to improve. Differentiated expectations.
– Schools should not be rewarded for closing the achievement gap when it is
accomplished by lowering the score of the higher performing group.
10
Growth and School Type
A thoughtful approach should consider how standards interact with status.
Status/Growth Combinations
High Status
High/Low
Low/Low
High/High
Low/High
Low Status
Low
Growth
High
Growth
11
Build-In Incentives
• Consider ways to incentivize actions to support lowest
performing students
• For example, what will be the motivation to work with
students who don’t graduate in four years?
• Consider ‘incentive’ points
Possible Outcomes
Student earns CCR
diploma with honors
Student earns CCR
diploma
Student earns non-CCR
(e.g. ‘general’) diploma
Points - A
125
100
75
Additional
Outcomes
Student earns
(or is likely to
earn) college
credit via
AP/IB/dual
enrollment
Points - B
10 per
course
Time to
Outcome
Points - C1
4th year
NA
(sum of A
and B)
multiplied
by .75
5th year
Student earns GED
50
Student earns other
Student earns
certificate
50
10 per
industry
Student persists, does
(sum of A
certificate
certification
not meet graduation
and B)
6 + years
requirements
25
multiplied
by .50
Student drops out
0
1Late graduates are not averaged into the index score for the current cohort. However, bonus
points are added to the current cohort score after averaging.
12
Performance Expectations
• Is it important to establish criteria (whether
reported or not) within each component, or
should criteria be based only on overall results?
• Are there external indicators that should factor
into decisions?
• Are there expectations for the distribution of
results?
• How should expectations differ for schools/
subgroups?
13
Normative
• Normative growth provides an indication of
what is attainable
• Examine patterns of performance for schools
and subgroups to set initial expectations for
what is possible and what is reasonable
– For example: What is rate of growth for high
achieving students compared to low achieving
students?
14
Criterion
• For students that grow at specified rates, what
is the probability of attaining or maintaining
target status?
• For example:
– What percent of non-proficient students meeting
a specified growth target reach proficiency in 1
year, 2 years etc.
15
Growth Expectations
• Both are important and
complimentary
• Ideally, the are used
iteratively
– Examine patterns of
performance for schools and
subgroups to set initial
expectations for what is
possible and what is
reasonable
– For students that grow at
specified rates, what is the
probability of attaining or
maintaining target status?
Example depiction of norm and criterion
referenced growth based on SGP. Betebenner,
D. W.
16
Evidence the System is Working
• To what degree are outcomes stable for schools/ groups of
various types and size? (reliability)
• To what extent are the results associated with variables not
related to effectiveness? (e.g. percent ED)
• What evidence bolsters the claim that classifications are
credible? (e.g. related to other valued quantitative and
qualitative indicators not modeled)
• Are the results useful for improvement?
• Are negative consequences mitigated?
Research is ongoing and should shape both initial design
and ongoing refinements
17