Contingencies for the use of effective
educational practices: Developing
Utah’s Alternate Assessment
Tim Slocum
Wing Institute First Annual Summit
April 2006
Main Topics


Contingencies and evidence-based
special education
Designing an alternate assessment
that can bring powerful NCLB
contingencies to bear on promoting
evidence-based special education.
Contingencies and EvidenceBased Practices
Contingencies

A contingency is a relationship between
behavior and consequences.


Many contingencies are “natural” - that is, not
intentionally constructed



If a certain behavior occurs, then a certain
consequence will occur.
Physical – Effects of jumping out of a window on falling
Social – Effects of speaking unclearly on interaction
Other contingencies are intentionally
constructed to influence behavior



Parenting – Family rules
Education – Classroom motivation systems
Legislation and policy – Laws and regulations
Contingencies and
Evidence-Based Practices


When we consider factors that may
promote or undermine a culture of
evidence-based practices, we should
consider contingencies that may make
such a culture more or less relevant.
What legislative and policy
contingencies provide a context for
evidence-based practices?
Effects of Contingencies

Contingencies can be very powerful for
changing behavior.


When powerful reinforcers are given contingent
on a particular desirable behavior, changes in
that behavior can be dramatic.
However, if contingencies are not carefully
constructed, they can produce unintended
outcomes.

A teacher may offer a reward contingent on
increased oral reading rate. This may result in
increased oral reading rate, but also increased
errors.
Contingencies – Defining Targets

There are two ways to define behaviors in a
contingency:

By their form – specifying what must be done:

Examples




Study 2 hours per night…
Reading First – a federal program that specifies forms of
behavior (research-based reading instruction) necessary to get
a reward (funding).
This requires a specific form of behavior but leaves the
outcome unspecified.
Unintended outcomes can come from completing the
behavior in ways that do not produce the intended outcomes.


Consider all the ineffective “study behaviors” that a student
might engage in.
Consider how mandated processes (IEP, functional analysis,
implementation of research-based practices) become formalized
and fail to achieve intended purposes.
Contingencies – Defining Targets

By their results – specifying what must be
achieved:

Examples




Get a GPA above 3.75…
No Child Left Behind – a federal program that specifies
outcomes (AYP) necessary to avoid punishment (public
scorn, loss of control, loss of funding)
This specifies what is to be accomplished but leaves
the process unspecified.
Unintended consequence can come from ways of
producing the narrowly defined outcomes without
producing the intended outcomes.


Getting a high GPA by taking easy classes
Raising test scores by excluding lower performing
students.
Contingencies


So, whether we aim at what a person must
do (form) or what they must accomplish
(achieve), we must be very careful about
defining what constitutes fulfilling the target
of the contingency.
When contingencies specify difficult
behaviors and promise strong
consequences, the chances of unintended
consequences are most serious.
Contingencies and NCLB

We can understand NCLB as a set of
contingencies that specify results but leave
specific forms of behavior unspecified.

NCLB has produced a lot of behavior oriented
toward achieving adequate yearly progress
(AYP).

Some of this behavior is desirable:



Searching for and implementing evidence-based
practices,
Focus on effectively teaching core subject matter.
Some of this behavior is undesirable:



Implementing narrow test-preparation programs,
Excluding lower performing students,
Reduced focus on important outcomes that do not affect
AYP.
Contingencies and
Evidence-Based Practices

When we talk about promoting an evidence-based culture in
special education, we must consider contingencies.

Contingencies that demand effectiveness provide a context for
an evidence-based culture.


In the absence of such contingencies, an evidence-based culture
is likely to be less relevant.



With such contingencies, many of the most serious barriers to
implementation of evidence-based practices are reduced or
eliminated.
Effective implementation of evidence-based practices requires
substantial effort from many players.
In the absence of strong contingencies, this kind of effort is unlikely.
When we think about each part of the Wing model of evidencebased education, we should consider the contingencies for key
actors at each point.
Alternate Assessment
Contingencies and
Alternate Assessment


NCLB is designed to include all students in a
school’s AYP – and that is a very good thing.
Given the strength of NCLB’s contingencies,
behavior that is not relevant to meeting its demands
is likely to be reduced in value.




Exclusion from contributing to AYP does not just imply
avoiding the pain and hassle of test-taking.
Exclusion makes the student’s educational progress
irrelevant to the powerful contingencies of NCLB.
Those who are not counted, don’t count.
NCLB includes a provision specifying that students
with significant cognitive disabilities (SCD) take an
alternate assessment that will contribute to AYP.
Utah’s Alternate Assessment

We (Karen Hager and I) became interested in
whether an alternate assessment could be
constructed that would take advantage of the strong
contingencies of NCLB to:




motivate effective educational practices and
improve educational outcomes of students with SCD.
We worked with the State of Utah to attempt to
construct an effective alternate assessment.
We realized that validity of the alternate
assessment would be critical.


Validity is a technical term for the degree to which an
assessment successfully accomplishes the purposes for
which it is being used.
Validity is critical for producing intended outcomes and
avoiding unintended outcomes from contingencies.
Importance of Validity

The contingency of NCLB is based on
test scores, not important student
learning per se.


To the degree that alternate assessment
results are valid, the contingencies of NCLB
could have very positive impacts.
To the degree that alternate assessment
results are invalid, the contingencies of
NCLB could be very detrimental.
Defining the Construct
We began with the question, “What are we
trying to measure?”

Federal mandate – AA must assess:


Our values – AA should assess:


Domains of Language Arts and Math for students with
SCD.
Skills that will contribute to independence and
effectiveness of students with SCD.
The construct that drove the development of Utah’s
Alternate Assessment:

Language Arts and Math skills that will contribute to
students’ independence and effectiveness for students with
SCD.
Sources of Invalidity

There are 2 main sources of invalidity:

Construct-Irrelevant Variance – the assessment
may be influenced by factors that are not part of
the target construct.


May result in strengthening the wrong behaviors.
Construct Under-representation – the
assessment may ignore some important
component of the construct.

May result in failing to strengthen some appropriate
behaviors.
Construct Irrelevant Variance

Sources of Construct Irrelevant Variance

Direct assessment of factors other than student
behavior


Assessment of skills in such a way that outcomes
are heavily influenced by how the teacher
documents student behavior.


E.g., academic skills that are not functional
Problems with administration


E.g., some forms of portfolio assessment
Inclusion of skills that are not important.


E.g., degree of inclusion, attendance, and other issues
E.g., Low fidelity of implementation and low reliability
Trivial tasks

E.g., assessments on which all students will be “proficient”
no matter what their skills.
Construct Irrelevant Variance

Implications of Construct Irrelevant
Variance


These are sources of unintended effects
of the contingency
The contingency may strengthen
behaviors relevant to these other factors
at the expense of actually improving the
learning of important skills.
Construct Under-representation

Sources of construct underrepresentation

Some important skills are not assessed.


E.g., actual differences in critical
communication skills do not impact test
results
Implications of under-representation

The powerful contingencies of NCLB
may reduce focus on these important
skills.
Construct Under-representation


NCLB appears to be very powerful in
changing behavior of teachers and
administrators.
If this power is to benefit students with
significant cognitive disabilities, Alternate
Assessments must be highly valid.

That is, they must reflect actual student
performance on important tasks and must
exclude other factors as much as possible.
Design of Utah’s Alternate
Assessment
Standardization among diversity

The greatest challenge in designing an
alternate assessment is to:



Produce meaningful, standardized
outcomes
For a highly diverse group of students.
This requires a balance of
standardization and individualization.
Standardization in UAA

Standardization is necessary to enable
those not directly involved in the
administration to interpret scores.

Standardized bank of tasks


UAA includes approximately 115 tasks in language arts and
math
Standardized administration procedures



Each task requires direct observations of student performance
Tasks are to be embedded in naturally occurring routines in
natural settings
Any communication system or assistive technology the student
typically uses is acceptable during the assessment
Standardization in UAA

Performance criteria standardized for
each task:



Performance criteria are designed to reflect a
“functional” level of performance, i.e., a level that
makes the skill meaningful/useful
Students perform the task independently (e.g., no
physical, verbal, or gesture assistance)
Measurement tasks specify assessment of
generalization: Three performances in situations
that vary in relevant variables (e.g., people, settings,
materials)
Individualization in UAA

Individualization is necessary to ensure
relevance to specific students.

IEP team selects targets from list of
standardized tasks

IEP team selects content on some tasks

E.g., specific sight words to be assessed
Individualization in UAA

IEP team adapts the task to some
individual circumstances


IEP team specifies the natural routine during which to
assess
IEP team selects appropriate
generalization settings

E.g., particular individuals, settings, materials
Defining Adequate Yearly Progress

AYP for general education students


Standards set by grade level
AYP for students with SCD


Grade level not relevant basis for defining
standards
Tasks that define AYP must be identified
individually.


In UAA, IEP team selects relevant tasks from task bank.
Given a relevant task, performance level
necessary to declare AYP is standardized.
Example UAA Task Description
Task Description
Area
Language Arts
Task Name
Makes choices (2-3 concrete items) (212)
Task Description
When presented with 2 - 3 concrete objects, the student will select a
preferred object and show interest in his/her choice.
(Alternative communication and technology acceptable. See pg. XX.)
Example UAA Task Description
Administration and Scoring
Set-up
Identify times within the context of daily activities when the student
is given opportunities to make choices between 2-3 concrete objects
(e.g., snack choice, playground equipment for recess).
Generalization
Student must demonstrate this skill across 3 objects, 3 settings, and
3 people.
Instructions
Present 2-3 objects and ask the student, “Which one do you want?”
Example UAA Task Description
Examples of this
task
Scoring
 Presenting the student with a puzzle and a ball and letting
him/her make a choice for free time.
 Presenting the student with a slice of pizza, a hamburger, and a
sandwich and letting him/her choose lunch.
 Showing student a hairbrush, a toothbrush, and a wash cloth
and letting him/her make choice of what task to complete first.
1. When shown 2 or 3 concrete items student will select preferred
object. (Selects)
2. Upon access, he/she will show interest in the choice (e.g.,
playing with object he/she chooses, eating preferred food
choice). (Shows interest)
Example UAA Task Description
Acceptable
prompts
Unacceptable
prompts
None.
Verbal or physical prompts to encourage student to make a choice
(e.g., gesturing, looking, pointing; picking an object up and saying,
“Is this the one you want?”).
Example UAA Task Record
Assessment Record
Area:
Language Arts
Task
Name:
Makes choices (2-3 concrete items) (212)
To be completed by IEP Team in the Fall
Student Name
Date IEP Team selected task
Example UAA Task Record
To be completed during the testing period in the Spring
Trial 1
Trial 2
Trial 3
Date
Object
Setting
Person
Selects
Shows interest
Yes
Yes
No
No
Yes
Yes
No
No
Yes
Yes
No
No
Example UAA Task Record
To be completed during the assessment window in the Spring
Check
one
Achievement Level
Criterion
Level 4
Level 3
Level 2
Level 1
3 correct trials
2 correct trials
1 correct trial
no correct trials
- Substantial
- Sufficient
- Partial
- Minimal
Accessing UAA

The entire manual for Utah’s Alternate
Assessment and the Technical Manual
can be found at:

http://www.usoe.k12.ut.us/sars/Data.htm