CONFIDENTIAL
Children First
Focus on Accountability
Accountability Initiative
July 19, 2006
CONTENTS
• Accountability Initiative
–Vision and Overview
–Achievement Reporting and Innovation
System (ARIS)
• Accountability Initiative Implementation
Timeline
1
THE PROBLEM
Grade 4 ELA- 2001 to 2004 School Average
720.0
700.0
Avg. Test Score
680.0
660.0
640.0
620.0
Number of
Schools = 52
Number of
Schools = 82
Number of
Schools = 113
Number of
Number of
Schools = 189 Schools = 205
600.0
0
10
20
30
40
50
60
70
80
90
Mix of Free Lunch / Black or Hispanic /ELL/ Spec. Ed.
2
THE SOLUTION: A CYCLE OF CONTINUOUS IMPROVEMENT
Accountabilities /
Measures
• Progress Reports
Leadership / clear goals
• Quality Reviews
SCHOOLS
Enablers / Tools
Innovation /
evaluation
Adjustment
• Continuous learning and
support (e.g. Empowerment
Schools Intensive)
• Periodic assessments
• ARIS
3
INNOVATIONS IN ACCOUNTABILITY
Old accountability measurements
New accountability measurements
• Snapshot of performance
• Progress: what the school adds
• Movement only across single
• All amounts of improvement in
proficiency drivers- ignores
outcomes on high/ low ends
• Only grades schools with challenging
populations: SINI, SURR
• Incentives to focus on students least
student achievement
• Universal application: All schools
are graded: A, B, C, D, F
• Extra credit for gains by students
most in need
in need
4 Year Cycle of Consequences
• School improvement plan and target setting
• Leadership change
• School restructuring
• School closure
4
ACCOUNTABILITIES / MEASURES: SCATTER CHART
Scatter Plot of Peer Group Index Scores with Overall School Index Scores with and without Extra Credit
1.400
1.200
School Index Score
1.000
0.800
SCORETOTNOEC
SCORETOTEC1
Linear (SCORETOTNOEC)
Linear (SCORETOTEC1)
0.600
0.400
0.200
0.000
0
10
20
30
40
50
Peer Group Index Score
60
70
80
90
ACCOUNTABILITIES / MEASURES: QUALITY REVIEWS
• Understand what each
student knows and is
able to do monitor, and
• Measure the student’s
progress over time.
Gather data
Plan and
set goals
• Update
documentation /
best practices on
effectiveness of its
improvement plans
and flexibility of
change processes
Monitor
and revise
• Align instructional skills
and capacity development
around established goals
for accelerating individual
student learning
• Set each
student’s next
learning step.
• Collaborate with
student / parents
to set goals for
improving
teaching practice
and accelerating
student learning
Continuous
data-driven
improvement
Align
instruction
Build and
align
capacity
• Align school
instructional
activity and
resources around
focused plans for
accelerating
individual student
learning
6
ENABLERS / TOOLS: CREATE A CYCLE OF CONTINUOUS
LEARNING
Assess progress
and make midcourse corrections
Start by
analyzing school’s
performance data
to ID areas that
need improvement
Align
instructional
practice and
resources,
access external
resources to
support change
Identify high
leverage points
for intervention
to improve
student
performance
Top-down qualitative
analysis from Quality
Review
Create
urgency
with
publicized
results
Analyze studentlevel data,
instructional
practices, and
adult learning
needs
Publish
Progress
Reports
Deeply
informed
interventions
Bottom-up quantitative view
from Assessments
7
ENABLERS / TOOLS: A SET OF DIAGNOSTIC TOOLS-PERIODIC
PROGRESS MEASURES
8
ENABLERS / TOOLS: DIAGNOSTIC ASSESSMENT
METHODOLOGIES
October 2006
Jessica
Holloway
Edward
Smith
Raul
Fuentes
Jessica
Holloway
Edward
Smith
Raul
Fuentes
Summarizing
Main Idea
Fact vs. Opinion
Predicting
Context Clues
Paraphrasing
December 2006
Summarizing
Main Idea
Fact vs. Opinion
Predicting
Context Clues
Paraphrasing
9
ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS
THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION
School customized
assessments
Customized version
provided to align with
different curriculum,
sequence or desired
outcome (more predictive)
DOE and vendor
provided
assessments
Default system based
on citywide ELA and
Math curricula
• Math #1
• Math #2
• Math #7
• ELA #1
• ELA #5
School created
assessments
Periodic assessments
At every grade level, a set of
diagnostic assessment tools to
help inform interventions and
instruction to ensure an
individualized instructional plan for
every student
Standards-based
alternative assessment
system that..
• Is developed by
school, network or
region in
collaboration with a
partner
• Ensures equal rigor,
measures progress
and allows for crossgroup comparison
10
ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS
THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION
School customized
assessments
Customized version
provided to align with
different curriculum,
sequence or desired
outcome (more predictive)
DOE and vendor
provided
assessments
Default system based
on citywide ELA and
Math curricula
• Math #1
• Math #2
• Math #7
• ELA #1
• ELA #5
School created
assessments
ARIS
Standards-based
alternative assessment
system that..
• Is developed by
school, network or
region in
collaboration with a
partner
• Ensures equal rigor,
measures progress
and allows for crossgroup comparison
11
CONTENTS
• Accountability Initiative
–Vision and Overview
–Achievement Reporting and
Innovation System (ARIS)
• Accountability Initiative Implementation
Timeline
12
OBJECTIVES AND SCOPE OF ARIS
Scope of the ARIS
Objectives
• Provide principals, teachers and parents with
Ensure access to the information
and tools necessary to enable
online information on student achievement,
including periodic assessments
• Support development of longitudinal and
detailed analysis
• Develop an integrated portal, including a
dashboard to analyze key environment
factors and achievement metrics
• Enable real-time prediction of school
performance against year-end targets
• Generate standard reports for specific endusers
• Develop scorecards to enable drill-downs by
student, assessment, strand and sub-strand
analysis
• Provide knowledge management tools that
capture teaching and assessment content
generated at school level with an ability to
approve, publish and share with the
networks or the broader district. As well as
capturing new quality review data
• Longitudinal and detailed analysis
and reporting of achievement and
performance data
• Best-in-class assessment design
and implementation processes to
extract data from multiple sources,
including designing own reports
and the capability to slice and dice
the data
• Tools to improve learning of
students with a range of identified
needs;
• Sharing best practices and
collaborating across schools to
encourage cultures of continuous
school improvement and
professional learning
13
WHAT ARIS IS AND WHAT IT IS NOT
ARIS is…
• A solution that…
– Analyzes and reports
– Assesses
– Innovates (assessments, responsive
interventions)
– Collaborates and personalizes
– Shares knowledge (sharing) and
manages content
• A solution that may not be available offthe-shelf
• A solution that may require a
partnership or consortium of Bidders to
offer, not by a single vendor or a
service provider
• A collaborative effort between the
Vendor and NYCDOE to innovate,
develop and execute
ARIS is not…
• A system that will replace all of the current
applications now in operation at the DOE
(e.g. ATS, HSST). Users will still need to
update and print reports very much in the
same way they do today but the access to
all of the key systems will now all be well
integrated through a single interface
• An Enterprise Data Warehouse (EDW) that
will have all the current available data in
DOE (assessment, Financial, HR, etc.).
ARIS may be the foundation for an EDW in
the long run, but initially the data warehouse
will only contain student achievement data
and the necessary information to analyze by
several key criteria (e.g., age, school,
grade, background, etc.)
• An opportunity to replace desktops and/or
network infrastructure
14
OVERVIEW OF REQUIRED FUNCTIONALITY
1
2
Use best-in class assessments to
improve student achievement
Periodic assessments
• For school- or center-created
– Library of templates/idea bank
for school-created tests (e.g.,
rubric-based, standardized
multiple choice)
– Test creation and upload
• For 3rd party-created
– Assessment content upload
into online testing, reporting
and analysis
• For all
– Online testing and scoring
– Score data upload into
reporting and analysis
3
Generate reports and analyze
achievement/performance data
Analysis + Reporting
• Canned and customized reports
• Drill-down, roll-up, filtering
• Ad-hoc queries
• User-generated databases
• Adding new data to central
repository
• Data viewing in 3rd party tools
(e.g., Excel)
Advanced analysis
(for selected users)
• Advanced statistics
• Pattern detection for target setting
Innovate the professional
learning / share best
practices across all schools
Knowledge and plan
management
• Library of templates for
different document types (e.g.,
lesson plans, curriculums)
• Document creation and upload
to repository
• Document tagging
• Collaborative document
creation
• Publishing approval
• Document linking to reporting
(surveys)
Knowledge and plan management (continued)
• Searching documents in the repositories
• Document viewing and download
• Pattern analysis across text documents
Collaboration
• Posting announcements; setting alerts
• Online discussions; online messaging
• Custom workspaces, integration w/email
Personalization
• Pre-defined home page based on user role
• Personalization of home page (HP) by user
• Adding new tools to HP from a tool catalog
Auditing (ie.,usage instances, frequency, modification history tracking)
15
KEY TECHNICAL AND SYSTEM REQUIREMENTS
Context and challenges
• Data issues
– Significant archive size
– Uncertain data quality
– Critical data element gaps
• Legacy applications
– Significant number, fragmented
functionality
– Diverse but emerging
preferences for platforms
– Goal of having a single
Enterprise Data Warehouse
• Infrastructure
– Uneven / unclear quality of...
• Desktop standards /
availability
• Network access
– Stringent performance
requirements
– Funding constraints
Implications for ARIS RFP / requirements
• Data issues
Vendors, based on their experience, to work
with DOE on developing a processes and tools
for…
– Data cleaning and migration
– Filling critical data element gaps
– Improving data entry quality
• Legacy applications
Bidders to..
– Develop new data sources for new
accountability functions (e.g., QR)
– Offer a scalable and evolvable data model
– Comply with strongly preferred, but not
mandatory standards
• Infrastructure
– Bidders have the option to host externally, or
suggest hardware for DOE-hosted solutions
– Optimized for mobile/ existing infrastructure
– Funding for infrastructure outside of ARIS
contract scope
16
DOE ARIS PREPAREDNESS TASK FORCE FOR KEY TECHNICAL AND
SYSTEM REQUIREMENTS
Context
NYCDOE has set up
a Task Force to set
ARIS for success by
preparing internal
DOE systems and
processes to
complement vendor
solution
Objectives
The task force will…
• Data Integrity: Create solutions
• Identify level of cleanliness
and define processes to ensure
and maintain confidence in the
quality of data and validity of
ARIS outputs
• Infrastructure: Define the
of existing data1
• Identify key causes for
data inaccuracy or
incompleteness
infrastructural improvements for
maximizing usage and
effectiveness of ARIS
• Design and pilot solutions
• Data Source Inventory: Identify
• Recommend processes for
alternate types of approaches/
applications schools use to host
data
• Communications: Craft
messaging tailored to user
experience
to clean existing source
systems feeding ARIS2
maintaining data quality
going forward
• Concentrate on
transactional systems
(ATS, HSST, etc.)
17
( _ ) Segment size
OVERVIEW OF TRAINING APPROACH
Phase 2 (Sep. 2008)
Phase 1 (Sep. 2007)
Role-based
view
User
segments
Skill-based
view
•
•
•
•
•
•
•
•
•
Central/DAA (50)
Network / Regions (280)
Principals (1,400)
Other school achievement leaders1 (5,600)
Teachers (87,000)
Support staff / DIIT (?)
• Parents (~100s K)
Power (352)
Advanced (~4,3503)
Standard (~90,0004)
• Standard (~ 100s K)
• Initial training (role-based, project- and
Training
approach
1
2
3
4
•
site-based)
– One-on-one (for power and advanced)
– Train-the-trainer (for standard) - school
achievement leaders train other
teachers
Ongoing need-based training
– Phone support service
– Online simulation exercises
Includes select admin staff, select teachers
Includes 25 Central Staff/Accountability Office; 10 power users drawn from schools
Includes 150 network/region; 4200 principals and other school achievement leaders
Primarily teachers
• Initial training
– Train-the-trainer – select
volunteers to train others
• Ongoing need-based training
– Phone support service
– Online simulation exercises
RFP requires the vendors to
submit plans for measuring
training effectiveness
18
CONTENTS
• Accountability Initiative
–Vision and Overview
–Achievement Reporting and Innovation
System (ARIS)
• Accountability Initiative Implementation
Timeline
19
ACCOUNTABILITY INITIATIVE IMPLEMENTATION TIMELINE
2006-07
2007-08
Empowerment
Schools
Other Schools
All Schools
Progress Report (Previous Year)
Yes
No
Yes
Progress Report (Current Year)
Yes
Yes
Yes
School Targets
Yes
No
Yes
School Grade
Yes
Yes
Yes
Quality Review
Yes
Yes
Yes
Rewards/ Consequences
Yes
No
Yes
Spring 2007
No
Yes
Periodic Assessments
Yes
No
Yes
Support for Continuous Learning
Yes
No
Yes
Data Management System
20