Exploring Alternate AYP
Designs for Assessment and
Accountability Systems
Dr. J.P. Beaudoin, CEO, Research in Action, Inc.
Dr. Patricia Abeyta, Bureau of Indian Education
NIEA 2008 Annual Conference
Seattle, Washington
1
Session Goals
• Frame the technical and resource
considerations for decision-makers
• Provide clarification on the regulatory
provisions associated with NCLB
• Share assessment and accountability
design features
2
Guiding Questions
urpose
How does the purpose for the change/action
serve Native children?
irection
What direction will the change/action provide
the educational community?
otivation
To what degree will the change/action motivate
the Native peoples?
3
30,000ft. SYSTEM VIEW
Cultural
Student
Learning
Outcomes
Education
School
Family
Content &
Curriculum
Community
Instruction
School
Assessment
Accountability
4
School System Components
Academic Content Standards
“What do students need to know and be able to
demonstrate?”
• Instruction
“How is content organized, provided, modeled, and
expanded upon?”
• Assessment
“To what degree is the content expressed accurately
by the leaner?”
• Accountability
“Which learning indicators are being reached,
showing progress, or remaining unchanged?”
5
Integration
Things to Consider…
• Changes to one component will have some
influence on the remaining components
• Improvements in one area often result in
benefits throughout the system
• Isolated changes create bulkization,
inefficiencies, and internal stressors
6
Leverage
7
Compliance
If you change the
academic content
standards used for
AYP…
Then you must provide
evidence that addresses
Section 1 of the USDE’s
Peer Review Guidance
If you change the
assessments used
for AYP…
Then you must provide
evidence that addresses
Sections 2-7 of the USDE’s
Peer Review Guidance
If you change how
AYP is defined…
Then you must create an
AYP Workbook that
address 10 accountability
principles.
8
Overview: Standards and
Assessments Peer Review
Conditioned on what is changed, evidence must be presented
to the BIE and USDE Peer Reviewers…
Section 1-Content Standards
Section 2-Achievement Standards
Section 3-Coherent Assessment System
Section 4-Technical Quality
Section 5-Alignment
Section 6-Inclusion
Section 7-Reporting
9
Overview: AYP Workbook
Conditioned on what is changed, policy and empirical
evidence must be submitted for review by the BIE and
USDE…
Principle 1-Includes all
schools and districts
Principle 2-Includes all
students
Principle 3-Includes an
allowable method for AYP
determinations
Principle 4-Includes
annual accountability
decisions
Principle 5-Includes
subgroup accountability
Principle 6-Based on
academic assessments
Principle 7-Has additional
indicators
Principle 8-Separate
decision for reading and
mathematics
Principle 9-System validity
and reliability
Principle 10-Includes
participation rates
10
Overview: Review Process
Identify Areas of
Change
Submit Request
Receive
Authorization
Implement Changes
Address Findings
External Review
Implement Required
Changes
Resubmit Additional
Evidence
Submit
Implemen
tation
Evidence
Final Approval
11
ASSESSMENT
12
Generic Assessment Life Cycle
Examine
validity
evidence
Establish
assessment
purpose(s)
Report
results
Administer,
scan, score,
calibrate
Build test
specification
Create &
review
operational
forms
Develop
items, field
test, & review
items
13
Foundational Components
 Academic content standards
 Articulate what student are expected to know
and be able to do
 Grade level expectations in reading/ELA and
mathematics
 Grade span clusters in science
 Contain skill and process knowledge
14
Foundational Components
 Academic achievement standards
 Performance level descriptors
 No less than three achievement levels
(with applicable labels)
 “Cut-Score” established using a
technically valid standard-setting
approach (e.g, Modified Bookmark, Body
of Work, Teacher Judgment)
 Established by subject matter experts
(teachers) and other panelists with
applicable skills
15
Foundational Components
 Assessments aligned to intended
purposes
 Assessments in grades 3-8 and high
school:
 Reading/ELA
 Mathematics
 Science (3 grade clusters)
 Standardized administration and
scoring procedures
16
Foundational Process
 Typically 18-24 month development




process
Documented evidence
Reviewed by external (to USDE) experts
Evaluated evidence against Peer Review
Guidance
Resubmissions typical
17
USDE PEER REVIEW
GUIDANCE
Standards and Assessments
18
Section 1: Academic Content
Standards
Students know and are able to…
 Content standards are:
 Adopted in mathematics; reading/ELA
 Adopted in science (grade spans)
 Coherent and rigorous
 Developed by stakeholders
19
Section 2: Academic
Achievement Standards
The level students have mastered the content is…
 Achievement standards are:
 Adopted in mathematics, reading/ELA, and
science
 Assigned achievement levels and contentbased descriptors
 Established using an appropriate standardsetting procedure
 Aligned to the content standards
20
Section 3: Coherent Assessment
System
The different assessment components within the larger
assessment system contribute…
A coherent system:
 Is based on state and/or local assessments
 Provides integrated achievement data
 Aggregates only comparable measures
 Utilizes multiple measures focused on
higher order thinking skills
 Implements alternate assessments
21
Section 4: Technical Quality
The technical quality of the assessments is…
High quality assessments are:
 Valid measures based on their purpose
 Constructed to measure the content standards
 Accurate and consistent
 Fair and accessible
 Consistent across different forms and events
 Standardized in their administration and scoring
22
Section 5: Alignment
The assessments reflect the content standards characteristics
by…
Strong alignment is provided by using:
•
Using procedures to improve alignment
•
Techniques to match the content standards and patterns of
emphasis
•
Capture the range and depth of knowledge in the standards
•
Pattern of emphasis
•
Approaches to measure both content and process
knowledge
•
Content-based performance descriptors
23
Section 6: Inclusion
The assessment system includes all students by…
Students participate in the assessments as
demonstrate by:
 Impact data showing high rates of student
participation
 Affording students with disabilities participation
under accommodated conditions
 Accommodating students learning to speak
English
 Including migrant and other highly mobile
students
24
Section 7: Reporting
The assessment reports provide accurate and timely
information about student achievement by…
Student achievement data is:
 Reported using valid and credible
procedures
 Based on all students from valid assessments
 Reported at the individual student,
subgroup, school, district, and state-levels
 Disseminated in a timely manner
 Easily understood by parents and the public
25
USDE REVIEW GUIDANCE
AYP Workbook
26
Principal 1: Includes All
Schools and Districts
 Includes all schools and districts
 Holds all schools to the same criteria
 Incorporates the academic
achievement standards
 Provides information in a timely
manner
 Includes report cards
 Includes rewards and sanctions
27
Principal 2: Includes All
Students
 Includes all students
 Consistently defines full academic
year (FAY)
 Includes mobile students
28
Principal 3: Method of AYP
Determinations
 Applies universal proficiency by 2013-
2014 as the long-term goal
 Uses a method to make subgroup,
school, and district AYP
determinations
 Establishes a starting point with
statewide annual measurable
objectives (AMOs)
 Establishes intermediate goals
29
Principal 4: Includes Annual
Decisions
 Determines annually the progress
of schools and districts
30
Principal 5: Includes
Subgroup Accountability
 Includes all the required student subgroups
 Holds schools and LEAs accountable for the
progress of student subgroups
 Includes students with disabilities (SWD)
and limited English proficient (LEP)
students
 Includes an established minimum n-count
 Includes safeguards to protect student
privacy
31
Principal 6: Based on
Academic Assessments
 Based primarily on academic
assessments
32
Principal 7: Includes
Additional Indicators
 Includes graduate rate for high
schools
 Includes an additional academic
indicators for elementary and middle
schools
 Establishes valid and reliable
additional indicators
33
Principal 8: Includes Separated
Decisions for Reading/ELA and Math
 Holds students, schools, and
districts separately accountable for
reading/ELA and mathematics
34
Principal 9: System Validity
and Reliability
 Produces reliable decisions
 Produces valid decisions
 Addresses changes in assessments
and student populations
35
Principal 10: Includes
Participation Rates
 Includes a way to calculate the rate
of participation on the statewide
assessments
 Applies the 95% participation
criteria to student subgroups and
small schools
36
Research in Action, Inc.
World-Class Educational Quality™
Voice: 1.225.571.2408
E-Mail: ria2001@eatel.net
Fax: 1.225.644.8472
Website: www.ria2001.org
37