Natick Public Schools
21st Century Teaching and Learning Study:
Evaluation Results
Damian Bebell
James Burraston
Center for the Study of Testing, Evaluation, &
Educational Policy
Lynch School of Education
Boston College
Dec 15, 2014
Background and perspective
• Empirical educational research, measurement, and evaluation
• Background in psychometrics, measurement, and evaluation
• Leveraging technology to advance the state of the art in
educational research and measurement…
Data for the sake of data
vs
Ownership of “success”
and ongoing formative
use of information
Quantifying computer access
US Average Student : Computer Ratios
1983 = 125 : 1
1998 = 6 : 1
2007= 4 : 1
2010=3 : 1
Source: Market Data Retrieval, Public School Technology Survey 1998-2006; Education Week, 2007; Gray, Thomas, & Lewis, 2010
The rise of 1:1 Computing
Shared access
Sporadic use
Limited impact
1:1 Computing
•
•
•
•
•
1:1 computing programs continuously evolving
Widespread proliferation across world
Few programs have more than 5 years experience
Fewer still have invested in empirical research and evaluation efforts
So, lots of interest but limited research and evidence
1:1 student computing research
•
1:1 defines critical technology access…pedagogical
inferences
•
Emphasis on selection and mechanics of program, more
limited understanding of how teaching and learning change
•
Building level leadership
•
Pedagogical and content support for teachers
•
Systematic reflection, asking questions, and willingness to
adapt/evolve
Many different definitions of success
Natick/Boston College Research Partnership
September 2012- June 2014
Design and conduct external study to document impacts and
changes as students and teachers transition to new technology
rich computing environments.
Document through a series of reports and presentations
potentially evolving teaching and learning practices afforded by
the rich learning settings.
Document how Natick teachers and students accessing, using,
and relating to technology in support of teaching and learning.
Provide NPS with a rich empirical model (and data) for all future
inquiries and investigations concerning the relationship between
technology practices and a variety of outcomes.
www.bc.edu/natick21
Natick/Boston College Research Partnership
Acknowledgments
School Leadership
Teachers
Students
District Leadership
Natick Community
www.bc.edu/natick21
Long Term Goals/Outcomes
Student Outcomes/Program Goals
• Decrease behavioral and attendance issues
• Decrease special education/ACT referrals
• Increase in students’ computer use time in all subject areas
• Increase in student use of computers for doing homework in all subject areas
• Increase in homework completion
Staff/Teacher Outcomes/Program Goals
• Increase in teachers use of technology-based resources to enhance instruction
• Increase in teachers allowing students to use multiple technology-based resources
to address assessment demands
• Increase in the frequency that teachers structure learning opportunities that: are
inquiry-based, include a writing component, allow students to engage with and
utilize technology-based resources, and allow students to demonstrate mastery
• In Year 1 (2012/2013), revision of at least one unit per trimester where technology
is used to transform learning from a traditional teacher-directed curriculum-based
activity to a blended learning experience with an inquiry-based mode of facilitating
learning
• Decrease the number of teacher complaints concerning the lack of technology
resources available for instruction
NHS Research/Evaluation Design
• 7th-12th Student Survey (*Fall 2010, Fall 2012,
Spring 2013)
*92% of the NPS seventh grade student population participated in a pilot
survey in Fall 2010.
• 7th-12th Teacher Survey (Fall 2012, Spring 2013)
• Classroom Observations
• Focus Groups
• Staff Interviews
• Secondary analyses of school records
(achievement, attendance, Special Ed status, course patterns)
• Student Video Competition
NHS Student Video Competition
“How have computers and
technology changed the way you
learn new things and/or how your
teachers teach.”
www.bc.edu/natick21
Survey Response Rates/Parent Consent
So, what did we learn?
Selected Results
•Intro to Results
•Dramatic increase in student tech use/shift in
teacher practices
•Exploring trends within the student results
•Student and Teacher Beliefs
•Exploring Quantitative Impacts and Outcomes
Results:
Students’ increased use of
technology across grades and
subject areas
Major increase in students’ frequency of technology use
across core classes (2010-2012):
Major
shifts in
students’
classroom
practices
(2010-2012)
Major shift in teacher’s frequency of technology
use in the classroom (2010-2012):
Shifts in teachers’ classroom computer use
according to students (2012-2014)
Where (and how often were) NHS students using
technology in 2014?
Digging deeper in the Results:
Exploring trends within
student practices
(2013-2014)
Exploring gender
differences in
frequency of NHS
boys and girls
technology use
across their classes
Exploring differences in student and teacher technology use across
NHS students enrolled in one or more honors courses
Exploring differences in student practices comparing NHS students
enrolled in one or more honors courses
Exploring differences in
student practices across
socio-economic status for
NHS students
Results:
Natick Student and Teacher
Attitudes and Beliefs
Natick students report using a computer in
school improves their education:
Natick teachers report using a computer in
school improves their teaching:
Results:
Empirically examining student
outcomes associated with 1:1
Computing and new facilities
Impacts on NHS student homework completion
Average number of minutes completing home per night and the percent of homework
completed using a computer
Impacts on NHS student homework completion
Average homework completion rates self-reported by students
Impacts on NHS Student Attendance
Average student absences before and during first year 1:1 implementation
Impacts on NHS Student Attendance
Distribution of student absences before and during the first year of 1:1 computing
There appears to no relationship between students reported frequency of
technology use in school and their absences
Impacts on NHS Student Discipline
Paired-sample t-tests examining change in frequency of detentions from the
2011-2012 school year to 2012-2013.
Detentions
for
electronic
devices
0.06
-0.09
0.1
0.07
524
524
523
523
2.01
-4.22
0.045*
<0.001**
Students
# of Detentions
with detentions for skipping
detentions per student
school
Mean difference
Pearson’s r
# of students
df
t
p (two-tailed)
-0.06
0.42
524
523
-3.2
0.001**
0.12
0.51
524
523
-1.79
0.075
Impacts on NHS Student Discipline
Independent-samples t-tests comparing students technology-related
disciplinary problems from 2012/2013 to 2013/2014 across grade levels
Grade
8
9
10
11
12
Mean # of students # of students
change 2012-2013 2013-2014
-0.68
326
321
-1.1
346
290
-0.67
281
258
-0.12
221
218
-0.79
59
159
df
t
p (twotailed)
643
626
536
435
98
-2.01
-4
-2.36
-0.41
-1.54
0.04
0
0.02
0.68
0.13
Impacts on NHS Student Discipline
Average teacher and student perceptions of students' ability to get around the
school Internet filter
Impacts on NHS Special Education
Percentage of students who received various levels of special education services during
the last year in the old high school building and the first year in the new building.
Impacts on NHS student achievement
Average ELA and Math MCAS scaled scores over time for the NHS class of 2015 and 2016
Impacts on NHS student achievement
Percentage of students who scored "proficient" or better on the MCAS from
6th to 10th grade for classes of 2015 and 2016
Impacts on NHS student achievement
Percentage of students who scored as “proficient” or higher on the 10th grade
MCAS from 2009 to 2014.
Revisiting Long Term Goals/Outcomes
Student Outcomes/Program Goals
• Decreasing behavioral and attendance issues
• Decreasing special education/ACT referrals
• Increase in computer use time in all subject areas
• Increase in student use of computers for doing homework in all subject areas
• Increase in homework return rates
Staff/Teacher Outcomes/Program Goals
• Increase in the frequency that teachers use technology-based resources to enhance
instruction
• Increase in the frequency that teachers allow students to use multiple technologybased resources to address assessment demands
• Increase in the frequency that teachers structure learning opportunities that: are
inquiry-based, include a writing component, allow students to engage with and utilize
technology-based resources, and allow students to demonstrate mastery
• In Year 1 (2012/2013), revision of at least one unit per trimester where technology is
used to transform learning from a traditional teacher-directed curriculum-based activity
to a blended learning experience with an inquiry-based mode of facilitating learning
• Decrease the number of teacher complaints concerning the lack of technology
resources available for instruction
Questions/Follow Up
• Damian Bebell, Principal Investigator
• bebell@bc.edu
• James Burraston, Research Associate
• james.burraston.1@bc.edu
www.bc.edu/natick21