Final Report
A Second Year Evaluation Study of Promethean ActivClassroom
Prepared by Marzano Research Laboratory
for
Promethean, Ltd.
December, 2010
MARZANO RESEARCH LABORATORY
www.MarzanoResearch.com
Phone: 888-849-0851  Fax: 866-308-3135
OUR MISSION
To provide the best research, the
most useful actions, and the highest
level of services to educators.
OUR VISION
To continuously develop tools that
translate high quality educational
research into practical applications
educators can put to immediate
use.
OUR GOAL
To be the place educators go for the
latest information and data,
synthesized into clear, concise
resources that facilitate immediate
action.
REPORT AUTHORS
Robert J. Marzano & Mark W. Haystead
For citation purposes, please refer to this document as:
Marzano, R. J., & Haystead, M. W. (2010). Final report: A second year evaluation study of Promethean
ActivClassroom. Englewood, CO: Marzano Research Laboratory.
Copyright © 2010. All rights reserved. No part of this publication may be reproduced or transmitted in any form or
by any means without permission from Promethean, Ltd. http://www.prometheanworld.com.
BUSINESS DEVELOPMENT OFFICE
555 N Morton Street
Bloomington, IN 47404
Phone: 888-849-0851
Fax: 812-336-7790
RESEARCH CENTER
9000 E. Nichols Ave. Ste. 112
Englewood, CO 80112
Phone: 303-766-9199
Fax: 303-694-1778
www.MarzanoResearch.com
Table of Contents
Table of Contents ......................................................................................................................................... i
Table of Figures ......................................................................................................................................... iii
Executive Summary ................................................................................................................................... vi
Introduction ..................................................................................................................................................1
Phase I ..........................................................................................................................................................2
Design of the Evaluation Study for Phase I ...........................................................................................3
The Use of Meta-Analysis .....................................................................................................................3
The Sample ............................................................................................................................................4
Demographics ........................................................................................................................................7
Data Analysis and Findings .................................................................................................................11
Evaluation Question 1: What effect does Promethean ActivClassroom have on students’
achievement regarding the subject matter content taught by their teachers?.................................12
Evaluation Question 2: Does the effect of Promethean ActivClassroom differ between school
levels? ............................................................................................................................................20
Evaluation Question 3: Does the effect of Promethean ActivClassroom differ between grade
levels? ............................................................................................................................................21
Evaluation Question 4: Does the effect of Promethean ActivClassroom differ between academic
content areas? .................................................................................................................................23
Evaluation Question 5: Is there a relationship between the effect of Promethean ActivClassroom
and length of teaching experience, how long teachers have used Promethean ActivClassroom, the
percentage of time Promethean ActivClassroom was used in the classroom, and teachers’ selfreported confidence in their use of Promethean ActivClassroom? ................................................24
Phase I Summary and Interpretation ....................................................................................................35
Phase II.......................................................................................................................................................38
Coding of Videotapes ..........................................................................................................................40
Evaluation Question 1: What is the relationship between student engagement and positive versus
negative effects for the use of Promethean ActivClassroom? .......................................................41
Evaluation Question 2: What is the relationship between teacher IWB skills and positive versus
negative effects for the use of Promethean ActivClassroom? .......................................................43
Evaluation Question 3: What is the relationship between student IWB skills and positive versus
negative effects for the use of Promethean ActivClassroom? .......................................................44
Evaluation Question 4: What is the relationship between multiple student use of the IWB and
positive versus negative effects for the use of Promethean ActivClassroom? ..............................46
Evaluation Question 5: What is the relationship between student independent use of the IWB and
positive versus negative effects for the use of Promethean ActivClassroom? ..............................47
Final Report
i
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 6: What is the relationship between teacher use of IWB reinforcers and
positive versus negative effects for the use of Promethean ActivClassroom? ..............................48
Evaluation Question 7: What is the relationship between teacher use of learner response systems
and positive versus negative effects for the use of Promethean ActivClassroom?........................49
Evaluation Question 8: What is the relationship between teacher use of the IWB to represent
knowledge graphically or nonlinguistically and positive versus negative effects for the use of
Promethean ActivClassroom? ........................................................................................................51
Evaluation Question 9: What is the relationship between teacher previewing of content and
positive versus negative effects for the use of Promethean ActivClassroom? ..............................52
Evaluation Question 10: What is the relationship between teacher chunking of content and
positive versus negative effects for the use of Promethean ActivClassroom? ..............................53
Evaluation Question 11: What is the relationship between teacher scaffolding of content and
positive versus negative effects for the use of Promethean ActivClassroom? ..............................55
Evaluation Question 12: What is the relationship between teacher pacing of content and positive
versus negative effects for the use of Promethean ActivClassroom? ............................................56
Evaluation Question 13: What is the relationship between teacher monitoring of student progress
and positive versus negative effects for the use of Promethean ActivClassroom?........................57
Evaluation Question 14: What is the relationship between clarity of content as depicted in the
IWB and positive versus negative effects for the use of Promethean ActivClassroom? ...............59
Evaluation Question 15: What is the relationship between student interaction about the content
and positive versus negative effects for the use of Promethean ActivClassroom?........................60
Evaluation Question 16: What is the relationship between student response rates and positive
versus negative effects for the use of Promethean ActivClassroom? ............................................61
Evaluation Question 17: What is the relationship between classroom management and positive
versus negative effects for the use of Promethean ActivClassroom? ............................................63
Interpretation ........................................................................................................................................64
Comparison of Means versus Bivariate Correlation ......................................................................64
Engagement and Student Achievement .........................................................................................64
Variables 6, 7, and 8 ......................................................................................................................65
Variables 10, 11, 12, 13, 14, and 16 ..............................................................................................65
Phase II Summary and Interpretation ..................................................................................................68
Technical Notes .........................................................................................................................................71
Appendix A – MRL Action Research Instructions ....................................................................................79
Appendix B – NCES Code Definitions .....................................................................................................82
Appendix C – Treatment/Control Study Data ...........................................................................................83
References ..................................................................................................................................................90
Final Report
ii
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Table of Figures
Figure 1. Participating Sites – Midwest Region ..........................................................................................5
Figure 2. Participating Sites – Northeast Region .........................................................................................5
Figure 3. Participating Sites – Southeast Region .........................................................................................5
Figure 4. Participating Sites – Southwest Region........................................................................................6
Figure 5. Participating Sites – West Region ................................................................................................6
Figure 6. Number of Participating Sites by School Level ...........................................................................6
Figure 7. Number of Students by Grade Level ............................................................................................7
Figure 8. Population Statistics and Locale Codes for Participating Sites ....................................................8
Figure 9. Number of Students by Study Locale Category – City ................................................................9
Figure 10. Number of Students by Study Locale Category – Suburb .........................................................9
Figure 11. Number of Students by Study Locale Category – Town ............................................................9
Figure 12. Number of Students by Study Locale Category – Rural ..........................................................10
Figure 13. Number of Students by Demographic Category – Gender .......................................................10
Figure 14. Number of Students by Demographic Category – Ethnicity ....................................................10
Figure 15. Number of Students by Demographic Category – Free & Reduced Lunch .............................11
Figure 16. Number of Students by Demographic Category – English Language Learner ........................11
Figure 17. Number of Students by Demographic Category – Special Education .....................................11
Figure 18. ANCOVA Findings for Treatment/Control Studies .................................................................13
Figure 19. Overall Random Effects ...........................................................................................................16
Figure 20. Homogeneity Analysis .............................................................................................................17
Figure 21. Distribution of Effect Sizes – Continuation Study ...................................................................18
Figure 22. Distribution of Effect Sizes – Combined Continuation and First Year Studies .......................19
Figure 23. Random Effects – School Level ...............................................................................................20
Figure 24. Homogeneity Analysis – School Level ....................................................................................20
Figure 25. Random Effects – Grade Level ................................................................................................21
Figure 26. Homogeneity Analysis – Grade Level .....................................................................................22
Figure 27. Random Effects – Academic Content Area..............................................................................23
Figure 28. Homogeneity Analysis – Academic Content Area ...................................................................24
Figure 29. Distribution of Teaching Experience – Continuation Study ....................................................25
Figure 30. Distribution of Teaching Experience – Combined Continuation and First Year Studies ........26
Figure 31. Distribution of Promethean ActivClassroom Use – Continuation Study .................................27
Final Report
iii
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 32. Distribution of Promethean ActivClassroom Use – Combined Continuation and First Year
Studies ........................................................................................................................................................28
Figure 33. Distribution of Percentage of Class Time – Continuation Study .............................................29
Figure 34. Distribution of Percentage of Class Time – Combined Continuation and First Year Studies .30
Figure 35. Distribution of Confidence Ratings – Continuation Study ......................................................31
Figure 36. Distribution of Confidence Ratings – Combined Continuation and First Year Studies ..........32
Figure 37. Product Moment Correlations for Teacher Survey Responses – Continuation Study .............33
Figure 38. Product Moment Correlations for Teacher Survey Responses – Combined Continuation and
First Year Studies .......................................................................................................................................34
Figure 39. Percentage of Second Ratings the Same as First Rating and One Score Point from First
Rating .........................................................................................................................................................40
Figure 40. Comparison of Means for Student Engagement .......................................................................41
Figure 41. Bivariate Correlation for Student Engagement ........................................................................42
Figure 42. Comparison of Means for Teacher IWB Skills ........................................................................43
Figure 43. Bivariate Correlation for Teacher IWB Skills ..........................................................................44
Figure 44. Comparison of Means for Student IWB Skills .........................................................................44
Figure 45. Bivariate Correlation for Student IWB Skills ..........................................................................45
Figure 46. Comparison of Means for Multiple Student Use of the IWB ...................................................46
Figure 47. Bivariate Correlation for Multiple Student Use of the IWB ....................................................47
Figure 48. Comparison of Means for Student Independent Use of the IWB .............................................47
Figure 49. Bivariate Correlation for Student Independent Use of the IWB ..............................................48
Figure 50. Comparison of Means for Use of IWB Reinforcers .................................................................48
Figure 51. Bivariate Correlation for Teacher Use of IWB Reinforcers .....................................................49
Figure 52. Comparison of Means for Use of Learner Response Systems .................................................50
Figure 53. Bivariate Correlation for Use of Learner Response Systems ...................................................50
Figure 54. Comparison of Means for Use of the IWB for Nonlinguistic Representation .........................51
Figure 55. Bivariate Correlation for Use of the IWB for Nonlinguistic Representation ...........................52
Figure 56. Comparison of Means for Previewing of Content ....................................................................52
Figure 57. Bivariate Correlation for Previewing of Content .....................................................................53
Figure 58. Comparison of Means for Teacher Chunking ..........................................................................54
Figure 59. Bivariate Correlation for Teacher Chunking ............................................................................54
Figure 60. Comparison of Means for Teacher Scaffolding .......................................................................55
Figure 61. Bivariate Correlation for Teacher Scaffolding .........................................................................56
Figure 62. Comparison of Means for Pacing of Content ...........................................................................56
Figure 63. Bivariate Correlation for Pacing of Content .............................................................................57
Final Report
iv
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 64. Comparison of Means for Monitoring Student Progress ..........................................................58
Figure 65. Bivariate Correlation for Monitoring Student Progress ...........................................................58
Figure 66. Comparison of Means for Clarity of Content on the IWB .......................................................59
Figure 67. Bivariate Correlation for Clarity of Content on the IWB .........................................................60
Figure 68. Comparison of Means for Student Interaction .........................................................................60
Figure 69. Bivariate Correlation for Student Interaction ...........................................................................61
Figure 70. Comparison of Means for Student Response Rate ...................................................................62
Figure 71. Bivariate Correlation for Student Response Rate .....................................................................62
Figure 72. Comparison of Means for Classroom Management .................................................................63
Figure 73. Bivariate Correlation for Classroom Management...................................................................64
Figure 74. Product Moment Correlations between Variables 10, 11, 12, 13, 14, and 16 ..........................67
Figure 75. Weighted Meta-Regression Analysis with Corrected Effect Size as the Dependent Variable
and Variables 10, 11, 12, 13, 14, and 16 as Independent Variables ..........................................................68
Final Report
v
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Executive Summary
During the 2009/2010 school year, Marzano Research Laboratory (MRL) was commissioned by
Promethean Ltd. to conduct a second year evaluation study of the effects of Promethean ActivClassroom
on student academic achievement. The findings from the first year evaluation study conducted during
the 2008/2009 school year were reported in Final Report: An Evaluation Study of the Effects of
Promethean ActivClassroom on Student Achievement (Marzano & Haystead, 2009). The weighted
average effect size estimate (Cohen’s d) for 85 treatment/control studies was .37 and statistically
significant at the .0001 level (p < .0001). When corrected for attenuation, the weighted average effect
size (Cohen’s d) was .44 and statistically significant at the .0001 level (p < .0001).
The second year evaluation study was conducted in two phases. Phase I involved an analysis of student
learning with and without Promethean ActivClassroom as it relates to demographic information and
teacher self-reported experience with the technology. Phase II involved an analysis of student learning
with and without Promethean ActivClassroom as it relates to teacher behaviors as evidenced in
videotapes of teachers using the technology in their classrooms.
In the treatment group, teachers used Promethean ActivClassroom to augment their instructional
practices. In the control group, teachers used strategies and materials to facilitate instruction without the
use of Promethean ActivClassroom. Because students could not be randomly assigned to treatment and
control groups, the second year evaluation study employed a quasi-experimental design, referred to as a
pretest-posttest non-equivalent groups design. The pretest scores were used as a covariate to statistically
equate the students and partially control for differing levels of background knowledge and skill.
Phase I of the second year evaluation study attempted to answer the following questions through a metaanalysis of the treatment/control studies:
Evaluation Question 1: What effect does Promethean ActivClassroom have on students’ achievement
regarding the subject matter content taught by their teachers?
Evaluation Question 2: Does the effect of Promethean ActivClassroom differ between school levels?
Evaluation Question 3: Does the effect of Promethean ActivClassroom differ between grade levels?
Evaluation Question 4: Does the effect of Promethean ActivClassroom differ between academic content
areas?
Evaluation Question 5: Is there a relationship between the effect of Promethean ActivClassroom and
length of teaching experience, how long teachers have used Promethean
ActivClassroom, the percentage of time Promethean ActivClassroom was used
in the classroom, and teachers’ self-reported confidence in their use of
Promethean ActivClassroom?
The weighted average effect size (Cohen’s d) for 46 treatment/control studies was .34 and statistically
significant at the .01 level (p < .01). When the 46 effect sizes (Cohen’s d) were corrected for attenuation
and combined, the weighted average effect size (Cohen’s d) was .39 and statistically significant at the
.01 level (p < .01). No significant moderator effects (p < .05) were found for the following variables:
school level, grade level, and academic content area. No significant correlations (p < .05) were found
between the 46 effect sizes (Cohen’s d) for the treatment/control condition (i.e., use of Promethean
ActivClassroom) and the following variables: length of teaching experience (reported in years), how
Final Report
vi
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
long teachers have used Promethean ActivClassroom (reported in months), the percentage of time
Promethean ActivClassroom was used in the classroom, and teachers’ self-reported confidence in their
use of Promethean ActivClassroom. However, a significant correlation (p < .001) of .57 was found
between teachers’ self-reported confidence in their use of Promethean ActivClassroom and how long
teachers have used Promethean ActivClassroom (reported in months).
When the results from the 46 treatment/control studies were combined with 85 treatment/control studies
from the first year evaluation study of ActivClassroom, the weighted average effect size (Cohen’s d)
was .36 and significant at the .0001 level (p < .0001). When the 131 effect sizes (Cohen’s d) were
corrected for attenuation and combined, the weighted average effect size (Cohen’s d) was .41 and
significant at the .0001 level (p < .0001). The combined data indicated no significant moderator effects
(p < .05) for the following variables: school level, grade level, and academic content area. No significant
correlations (p < .05) were found between the 131 effect sizes (Cohen’s d) for the treatment/control
condition (i.e., use of Promethean ActivClassroom) and the following variables: length of teaching
experience (reported in years), how long teachers have used Promethean ActivClassroom (reported in
months), and teachers’ self-reported confidence in their use of Promethean ActivClassroom. However, a
significant correlation (p < .05) of .22 was found between the effect sizes and the percentage of time
Promethean ActivClassroom was used in the classroom. In addition, a significant correlation (p < .001)
of .50 was found between teachers’ self-reported confidence in their use of Promethean ActivClassroom
and how long Promethean ActivClassroom was used (reported in months), a significant correlation (p <
.001) of .45 was found between teachers’ self-reported confidence in their use of Promethean
ActivClassroom and the percentage of time Promethean ActivClassroom was used in the classroom, and
a significant correlation (p < .05) of -.21 was found between teachers’ self-reported confidence in their
use of Promethean ActivClassroom and the length of teaching experience (reported in years). It is useful
to note that the combined results across the two evaluation studies involved 4,913 students, 123 teachers,
73 schools, and 36 districts.
Phase II of the evaluation study sought to answer the following questions:
Evaluation Question 1: What is the relationship between student engagement and positive versus
negative effects for use of Promethean ActivClassroom?
Evaluation Question 2: What is the relationship between teacher IWB skills and positive versus negative
effects for use of Promethean ActivClassroom?
Evaluation Question 3: What is the relationship between student IWB skills and positive versus negative
effects for use of Promethean ActivClassroom?
Evaluation Question 4: What is the relationship between multiple student use of the IWB and positive
versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 5: What is the relationship between student independent use of the IWB and
positive versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 6: What is the relationship between teacher use of IWB reinforcers and positive
versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 7: What is the relationship between teacher use of learner response systems and
positive versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 8: What is the relationship between teacher use of the IWB to represent knowledge
graphically or nonlinguistically and positive versus negative effects for use of
Promethean ActivClassroom?
Final Report
vii
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 9: What is the relationship between teacher previewing of content and positive
versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 10: What is the relationship between teacher chunking of content and positive
versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 11: What is the relationship between teacher scaffolding of content and positive
versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 12: What is the relationship between teacher pacing of content and positive versus
negative effects for use of Promethean ActivClassroom?
Evaluation Question 13: What is the relationship between teacher monitoring of student progress and
positive versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 14: What is the relationship between clarity of content as depicted in the IWB and
positive versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 15: What is the relationship between student interaction about the content and
positive versus negative effects for use of Promethean ActivClassroom?
Evaluation Question 16: What is the relationship between student response rates and positive versus
negative effects for use of Promethean ActivClassroom?
Evaluation Question 17: What is the relationship between classroom management and positive versus
negative effects for use of Promethean ActivClassroom?
Again, Phase II involved an analysis of student learning with and without Promethean ActivClassroom
from the perspective of teacher behavior as evidenced in videotapes of teachers using the technology in
their classrooms. The general focus of Phase II was to determine the behaviors that differentiated those
teachers who obtained positive effects from Promethean ActivClassroom from those who did not.
Two types of analyses were performed for each of the 17 questions. One analysis separated the teachers
into two categories: those with positive effects and those with negative effects for use of Promethean
ActivClassroom. Membership in these groups was considered the independent variable and scores on the
variables for the 17 Phase II evaluation questions were considered the dependent variables. An
independent samples t-test was conducted comparing the mean for the group with positive effects and
the mean for the group with negative effects for each of the 17 Phase II variables. The second analysis
correlated scores on the 17 Phase II variables with the corrected effect sizes from Phase I.
When the results from the first and second year evaluation studies were combined, the correlations
between 15 of the 17 variables addressed in Phase II and the corrected effect sizes calculated in Phase I
were statistically significant at the .05 level (p < .05). The correlations would also be considered
significant at the .01 level (p < .01) for all 15 variables and at the .001 level (p < .001) for 12 of the 15
variables. Student IWB skills (Variable 3) and student independent use of the IWB (Variable 5) did not
have significant correlations. Of the 15 Phase II variables with significant correlations to effect size,
only classroom management (Variable 17) did not exhibit a statistically significant difference on the ttest. However, it should be noted that the t-test approached significance at the .05 level (t = 1.94, df =
98, p = .056). The correlations between 14 of the 17 Phase II variables and the corrected effect sizes
were greater than .30. Of those 14 variables, chunking (Variable 10), scaffolding (Variable 11), pacing
(Variable 12), monitoring (Variable 13), clarity of IWB (Variable 14), and student response rate
(Variable 16) exhibited correlations with corrected effect size that were greater than .60. A multiple
correlation of .789 (p < .0001) was found between 99 corrected effect sizes and these 6 variables.
Final Report
viii
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Introduction
During the 2009/2010 school year, Marzano Research Laboratory (MRL) was commissioned by
Promethean Ltd. to conduct a second year evaluation study of the effects of Promethean ActivClassroom
on student academic achievement (hereinafter referred to as the continuation study). The findings from
the first year evaluation study were reported in Final Report: An Evaluation Study of the Effects of
Promethean ActivClassroom on Student Achievement (Marzano & Haystead, 2009). The weighted
average effect size estimate (Cohen’s d) was statistically significant (d = .37, N = 85, p < .0001). When
corrected for attenuation, the percentile gain associated with the use of Promethean ActivClassroom was
17 (d = .44, N = 85, p < .0001). None of the teachers, schools, or districts received compensation from
Promethean or MRL, their participation was strictly voluntary.
In March 2009 an invitation was sent to 28 school districts in various parts of the country soliciting
volunteer teachers willing to participate in a second round of treatment/control studies as part of the first
year evaluation study. Due to the timing of the data collection, 40 treatment/control studies were not
included in the first year study. In November 2009 an invitation was sent to 40 school districts
throughout the country (including some districts in the March 2009 invitation) seeking volunteer
teachers for participation in the continuation study and an additional evaluation study focusing on
Promethean Learner Response Systems. The 40 second round treatment/control studies from the March
2009 invitation were combined with 6 treatment/control studies from the November 2009 invitation as
part of the continuation study. The specifics of the Promethean technologies available for use in the
classroom can be found at Promethean’s website (see http://www.prometheanworld.com).
Final Report
1
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Phase I
Final Report
2
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Design of the Evaluation Study for Phase I
At a basic level, the continuation study sought to determine the effect that Promethean ActivClassroom
had on students’ achievement regarding academic content taught by their teachers. In more specific
terms, the continuation study employed one primary independent variable, whether teachers used
Promethean ActivClassroom to augment their current instructional practices (referred to as the treatment
group) or did not use Promethean ActivClassroom to augment their current instructional practices
(referred to as the control group). The dependent variable was students’ knowledge of academic content
addressed during a unit of instruction. Secondary independent variables of interest—students’ school
level, students’ grade level, academic content area—were treated as moderators. Meta-analytic
techniques (explained below) were employed to determine if the effect differed between groups on those
moderators.
One aspect of this evaluation study worth mentioning is that each treatment/control study used a quasiexperimental research design. Teachers acted as their own control and collected pretest and posttest
scores for two groups of students—one treatment and one control. The pretest scores were used as a
covariate to statistically equate the students and partially control for differing levels of background
knowledge and skill. Analysis of covariance (ANCOVA) is often used when random assignment is not
feasible (see Technical Note 1). This approach was used to mitigate potential differences in teaching
experience, style, methodology, etc. as possible explanations for differences between treatment and
control groups. Unlike experimental designs, quasi-experimental designs lack random assignment.
Quasi-experimental designs are often used when it is not possible or practical to randomly assign
subjects to equivalent groups. Although ANCOVA was used to statistically equate students in terms of
prior achievement, without randomization arguments about causal relationships are severely weakened.
(For a more thorough discussion of the treatment/control study design and its implications, see
Technical Note 2.)
Teachers were instructed to teach a short unit (or set of related lessons) on a topic of their choice to two
groups of students—one treatment and one control. Detailed directions provided to teachers can be
found in Appendix A. Briefly though, instructional activities in both groups were to be as similar as
possible except for the fact that Promethean ActivClassroom was used in one group only (i.e., the
treatment group). At the elementary grades—particularly the lower elementary grades—if two intact
groups of students could not be identified teachers were instructed to teach two separate units to the
same group of students ensuring that the subject matter of each unit was as closely related as possible
(i.e., complementary topics).
The Use of Meta-Analysis
Meta-analytic techniques (see Hedges & Olkin, 1985; Lipsey & Wilson, 2001; Cooper, 2009) were
employed to aggregate the findings from the treatment/control studies using the statistical software
package Comprehensive Meta-Analysis (CMA), v2.2.050 (Borenstein, Hedges, Higgins, & Rothstein,
2005). Meta-analytic techniques are often used to combine the results of studies on a common topic.
Final Report
3
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
In the continuation study, a common class of interventions was used in all treatment classes (i.e., use of
Promethean ActivClassroom). However, the studies employed teacher-designed pre- and postassessments of student academic achievement across various grade levels and academic content areas
requiring different dependent measures. A cautionary note must be made here. As mentioned previously,
all the treatment/control studies involved in the continuation study employed intact groups (i.e., students
were not randomly assigned to treatment/control conditions). Although ANCOVA was used to
statistically equate students in terms of prior academic achievement based on teacher-designed preassessments, arguments about causal relationships are not nearly as strong as they might be if group
membership had been determined through random assignment (see Technical Note 2).
To combine studies that used different dependent measures, the results of each study are translated into
an effect size. While there are several different effect sizes, the one used in this report is the standardized
mean difference effect size (Cohen’s d). A standardized mean difference is the difference between the
treatment and control group mean scores expressed in standard deviation units. By convention, a
positive effect size favors the treatment group. (For a discussion of effect size, see Technical Note 3.)
Meta-analytic findings are typically reported in two ways:
1. Weighted mean effect sizes computed from estimates of effect size for each treatment/control
study.
2. Weighted mean effect sizes computed from estimates of effect size for each treatment/control
study that have been corrected for attenuation due to lack of reliability in the dependent measure
(i.e., teacher-designed assessments of student academic achievement).
Technical Note 4 explains the method used to correct for attenuation and an interpretation of such
corrections. It is commonly recommended that estimates of effect size should be corrected for
attenuation due to unreliability of the dependent measure (for a detailed discussion of attenuation, see
Hunter & Schmidt, 2004). In basic terms, every assessment is imprecise to some extent. This
imprecision lowers the estimate of the true effect size. Throughout this report, observed and corrected
effect sizes are typically displayed for comparison. When this is the case, the discussion of findings is
limited to the corrected results only.
Although meta-analytic techniques were used to analyze most of the data, when more appropriate, the
general linear model was employed using the statistical software package, PASW® Statistics (SPSS),
v17.0.2 (SPSS, 2009). Specifically, when independent variables were more continuous in nature than
categorical, the general linear model was commonly employed.
The Sample
As Figures 1 through 5 illustrate, 27 public schools participated in the second year evaluation study from
20 cities in 14 states. The sites are grouped by their location in one of five geographic regions—
Midwest, Northeast, Southeast, Southwest, and West. Figures 1 through 5 also show the number of
teachers involved in the treatment/control studies at each site.
Final Report
4
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 1. Participating Sites – Midwest Region
Site
City
State
# of Teachers
1
Orchard Lake
MI
1
2
St. Louis
MO
4
3
Evansville
IN
1
4
Evansville
IN
1
5
Evansville
IN
2
6
Evansville
IN
1
10
Total
Figure 2. Participating Sites – Northeast Region
Site
City
State
# of Teachers
7
Groton
CT
1
8
New York
NY
1
9
Geneva
NY
5
Total
7
Figure 3. Participating Sites – Southeast Region
Site
City
State
# of Teachers
10
Sarasota
FL
1
11
Sulphur
LA
7
12
Lake Charles
LA
3
13
Suwanee
GA
1
14
Cumming
GA
1
15
Cumming
GA
1
16
Vestavia Hills
AL
2
17
Vestavia Hills
AL
2
18
Orlando
FL
1
Total
19
Final Report
5
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 4. Participating Sites – Southwest Region
Site
City
State
# of Teachers
19
Round Rock
TX
1
20
Pflugerville
TX
1
21
Pflugerville
TX
1
Total
3
Figure 5. Participating Sites – West Region
Site
City
State
# of Teachers
22
Snoqualmie
WA
1
23
Madera
CA
1
24
Wasilla
AK
1
25
Palmer
AK
1
26
Palmer
AK
1
27
Willow
AK
1
Total
6
Figures 1 through 5 indicate the continuation study involved 27 sites. In the Midwest region, 10 teachers
participated at 6 sites; in the Northeast region, 7 teachers participated at 3 sites; in the Southeast region,
19 teachers participated at 9 sites; in the Southwest region, 3 teachers participated at 3 sites; and in the
West region, 6 teachers participated at 6 sites.
Figure 6 displays the number of participating sites by school level along with the number of students in
control and treatment groups.
Figure 6. Number of Participating Sites by School Level
School Level
# of Sites
Control N
Treatment N
Total N
Elementary School
(Grades K – 5)
18
497
530
1,027
Middle School
(Grades 6 – 8)
4
186
199
385
High School
Grades (9 – 12)
5
86
77
163
Total
27
769
806
1,575
Final Report
6
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
The first year evaluation study involved 3,338 students—1,622 students in the control group and 1,716
students in the treatment group (Marzano & Haystead, 2009). The continuation study involved 1,575
students—769 students in the control group and 806 students in the treatment group. Of those students
in the continuation study, 1,027 were at 18 sites that teach students at the elementary school level, 385
were at 4 sites that teach students at the middle school level, and 163 were at 5 sites that teach students
at the high school level.
For comparison purposes, teachers were asked to report the grade level(s) taught in their
treatment/control studies. Figure 7 depicts the number of control and treatment students for each grade
level in the second year continuation study. Two teachers did not report a grade level. In addition, three
of the high school teachers reported more than one grade. Therefore, student counts for those teachers
were excluded from Figure 7.
Figure 7. Number of Students by Grade Level
Grade Level
Control N
Treatment N
Total N
K
23
33
56
1
116
119
235
2
57
71
128
3
73
81
154
4
126
128
254
5
84
83
167
6
20
19
39
7
68
81
149
8
98
99
197
9
25
18
43
10
--
--
--
11
--
--
--
12
--
--
--
Total
690
732
1,422
The previous two figures (Figure 6 and Figure 7) indicate that 153 students were excluded from Figure
7—79 from the control group and 74 from the treatment group. Only student counts from
treatment/control studies which identified a single grade level were included in Figure 7.
Demographics
To determine what type of community in which each participating school is located, MRL used data
from the National Center for Education Statistics (NCES), a federal entity located within the U.S.
Department of Education’s Institute of Education Sciences (see http://www.nces.ed.gov/). MRL utilized
the NCES website to search its online databases for each of the 27 participating schools involved in the
second year continuation study. Specifically, a search was conducted using the NCES Common Core of
Final Report
7
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Data (CCD) “Search for Public Schools” webpage (http://nces.ed.gov/ccd/schoolsearch/). CCD public
school data is currently available from the NCES website for the 2008-2009 school year. MRL also used
the U.S. Census Bureau’s American FactFinder website to search for population statistics from the 2000
U.S. Census (http://factfinder.census.gov/).
Figure 8 displays the city population statistics for each site, along with the NCES locale code and
category assigned to that site. The locale codes are assigned based on the proximity of a site’s address to
an urbanized area (see http://nces.ed.gov/ccd/rural_locales.asp). A listing of NCES site code definitions
can be found in Appendix B.
Figure 8. Population Statistics and Locale Codes for Participating Sites
Site
Locale
Code
Locale
Category
City
State
Population
(2000 Census)
1
21
Suburb
Orchard Lake
MI
2,215
2
21
Suburb
St. Louis
MO
348,189
3
12
City
Evansville
IN
121,582
4
12
City
Evansville
IN
121,582
5
12
City
Evansville
IN
121,582
6
12
City
Evansville
IN
121,582
7
22
Suburb
Groton
CT
39,907
8
11
City
New York
NY
8,008,278
9
32
Town
Geneva
NY
13,617
10
13
City
Sarasota
FL
52,715
11
22
Suburb
Sulphur
LA
20,512
12
13
City
Lake Charles
LA
71,757
13
21
Suburb
Suwanee
GA
8,725
14
21
Suburb
Cumming
GA
4,220
15
41
Rural
Cumming
GA
4,220
16
41
Rural
Vestavia Hills
AL
24,476
17
41
Rural
Vestavia Hills
AL
24,476
18
12
City
Orlando
FL
185,951
19
21
Suburb
Round Rock
TX
61,136
20
21
Suburb
Pflugerville
TX
16,335
21
21
Suburb
Pflugerville
TX
16,335
22
31
Town
Snoqualmie
WA
1,631
23
13
City
Madera
CA
43,207
Final Report
8
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Site
Locale
Code
Locale
Category
City
State
Population
(2000 Census)
24
32
Town
Wasilla
AK
5,469
25
32
Town
Palmer
AK
4,533
26
32
Town
Palmer
AK
4,533
27
43
Rural
Willow
AK
1,658
Sources: From NCES Common Core of Data (http://nces.ed.gov/ccd/schoolsearch/) and U.S. 2000 Census
(http://factfinder.census.gov/) .
Figures 9 through 12 list the number of control and treatment students in the second year continuation
study for each of the 12 NCES urban-centric locale codes.
Figure 9. Number of Students by Study Locale Category – City
Locale Code & Category
Control N
Treatment N
Total N
# of Sites
11 – City, Large
23
23
46
1
12 – City, Midsize
149
133
282
5
13 – City, Small
55
66
121
3
Total
227
222
449
9
Figure 10. Number of Students by Study Locale Category – Suburb
Locale Code & Category
Control N
Treatment N
Total N
# of Sites
21 – Suburb, Large
170
178
348
7
22 – Suburb, Midsize
135
149
284
2
23 – Suburb, Small
--
--
--
--
Total
305
327
632
9
Figure 11. Number of Students by Study Locale Category – Town
Locale Code & Category
Control N
Treatment N
Total N
# of Sites
31 – Town, Fringe
29
23
52
1
32 – Town, Distant
102
124
226
4
33 – Town, Remote
--
--
--
--
Total
131
147
278
5
Final Report
9
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 12. Number of Students by Study Locale Category – Rural
Locale Code & Category
Control N
Treatment N
Total N
# of Sites
41 – Rural, Fringe
94
95
189
3
42 – Rural, Distant
--
--
--
--
43 – Rural, Remote
12
15
27
1
Total
106
110
216
4
As Figures 9 through 12 illustrate, there are 449 students from 9 sites located in cities, 632 students from
9 sites located in suburban areas, 278 students from 5 sites located in towns, and 216 students from 4
sites located in rural areas.
Figures 13 through 17 list the number of control and treatment students in the second year continuation
study for each demographic category.
Figure 13. Number of Students by Demographic Category – Gender
Control N
Treatment N
Total N
Male
343
347
690
Female
325
345
670
Not Reported
101
114
215
Total
769
806
1,575
Figure 14. Number of Students by Demographic Category – Ethnicity
Control N
Treatment N
Total N
Asian
19
34
53
African American
81
85
166
Hispanic
54
55
109
Native American
3
5
8
Other
23
23
46
White/Caucasian
503
531
1,034
Not Reported
86
73
159
Total
769
806
1,575
Final Report
10
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 15. Number of Students by Demographic Category – Free & Reduced Lunch
Control N
Treatment N
Total N
Yes
220
219
439
No
409
431
840
Not Reported
140
156
296
Total
769
806
1,575
Figure 16. Number of Students by Demographic Category – English Language Learner
Control N
Treatment N
Total N
Yes
91
97
188
No
585
602
1,187
Not Reported
93
107
200
Total
769
806
1,575
Figure 17. Number of Students by Demographic Category – Special Education
Control N
Treatment N
Total N
Yes
70
58
128
No
604
636
1,240
Not Reported
95
112
207
Total
769
806
1,575
Data Analysis and Findings
One dependent variable was considered in the second year continuation study: students’ knowledge of
academic content addressed by their teacher during a unit of instruction. The primary independent
variable of interest was the treatment/control condition—whether students were exposed to Promethean
ActivClassroom or not. Also of interest were the differences in potential effect of Promethean
ActivClassroom with respect to three moderator variables—school level, grade level, academic content
area. The control condition represented the aggregate strategies and materials used by the teachers to
facilitate instruction without the use of Promethean ActivClassroom. Surveys submitted by participating
teachers indicated that they were quite diverse in the approaches they used.
As mentioned previously, two statistical software packages were utilized for data analysis. Data from
each treatment/control study was first analyzed with the general linear model using SPSS. One
independent variable (treatment/control condition) was entered into the equation using a fixed-effects
model (see Technical Note 5 for a discussion of fixed and random effects). The posttest scores were
entered as the dependent variable with the pretest scores used as a covariate. In other words, a fixedeffects ANCOVA was executed for each independent study. The ANCOVA findings were used to
compute observed and corrected effect sizes (Cohen’s d) for each study (see Technical Note 6 for the
formula used to compute the effect size). CMA was then used to aggregate the findings from the studies
Final Report
11
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
using the observed and corrected effect sizes for the treatment/control condition (i.e., use of Promethean
ActivClassroom).
The continuation study sought to answer the following evaluation question:
Evaluation Question 1: What effect does Promethean ActivClassroom have on students’ achievement
regarding the subject matter content taught by their teachers?
In addition, the following questions were considered through a meta-analysis of categorical variables:
Evaluation Question 2: Does the effect of Promethean ActivClassroom differ between school levels?
Evaluation Question 3: Does the effect of Promethean ActivClassroom differ between grade levels?
Evaluation Question 4: Does the effect of Promethean ActivClassroom differ between academic content
areas?
Finally, the following question was considered through a correlational analysis of teacher survey
responses:
Evaluation Question 5: Is there a relationship between the effect of Promethean ActivClassroom and
length of teaching experience, how long teachers have used Promethean
ActivClassroom, the percentage of time Promethean ActivClassroom was used
in the classroom, and teachers’ self-reported confidence in their use of
Promethean ActivClassroom?
Findings for each question are discussed separately.
Evaluation Question 1: What effect does Promethean ActivClassroom have on students’
achievement regarding the subject matter content taught by their teachers?
Figure 18 presents the findings and associated effect size (Cohen’s d) from the ANCOVA for 45
teachers involved in the continuation study. The columns labeled “Adjusted Mean” contain the posttest
mean adjusted for differences in the pretest scores for the control and treatment groups respectively
(number of students reported in parentheses). The column labeled “ES” contains the calculated effect
size (Cohen’s d) for each study (see Technical Note 6), the column labeled “Sig.” contains the p-value
(2-tailed) for each study, and the column labeled “% Gain” contains the percentile gain (or loss)
associated with each effect size. (For a discussion of effect size and associated percentile gain, see
Technical Note 7.)
Final Report
12
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 18. ANCOVA Findings for Treatment/Control Studies
Site
Teacher
Grade
Adjusted
Mean
(Control)
Adjusted
Mean
(Treatment)
ES
1
1
1
88.03
(n=21)
82.57
(n=20)
-.64
.0555
-23.9
2
2
4
78.94
(n=17)
74.29
(n=17)
-.30
.4077
-11.8
2
3
3
82.46
(n=17)
95.18
(n=17)
1.03**
.0072
34.9
2
4
4
74.71
(n=19)
54.76
(n=15)
-1.24**
.0017
-39.3
2
5
3
93.33
(n=16)
94.80
(n=16)
.19
.6104
7.5
3
6
3
68.16
(n=17)
63.74
(n=14)
-.50
.1982
-19.2
4
7
4
47.55
(n=20)
49.00
(n=19)
.09
.7889
3.6
5
8a
7
81.16
(n=22)
84.79
(n=26)
.31
.3108
12.2
5
8b
8
58.18
(n=20)
67.76
(n=15)
.77*
.0397
27.9
5
9
8
82.68
(n=21)
92.63
(n=17)
.96**
.0076
33.2
6
10
9
67.32
(n=25)
64.50
(n=18)
-.29
.3732
-11.4
7
11
1
63.70
(n=20)
70.43
(n=19)
.45
.1821
17.4
8
12
1
86.99
(n=23)
63.01
(n=23)
-1.65****
.0000
-45.1
9
13
8
46.22
(n=9)
83.82
(n=11)
2.54***
.0001
49.5
9
14
8
70.52
(n=17)
82.40
(n=17)
.45
.2237
17.4
9
15
7
71.10
(n=14)
85.86
(n=17)
1.81***
.0001
46.5
9
16
8
57.67
(n=13)
63.35
(n=18)
.39
.3148
15.2
9
17
7
77.68
(n=10)
81.01
(n=16)
.26
.5500
10.3
10
18
9-12
86.94
(n=2)
74.03
(n=4)
-1.70
.2600
-45.5
11
19
5
89.94
(n=14)
96.29
(n=17)
.61
.1172
22.9
Final Report
13
2009/2010 Second Year Promethean Evaluation Study
Sig.
(2-tailed)
% Gain
Marzano Research Laboratory
December 2010
Site
Teacher
Grade
Adjusted
Mean
(Control)
Adjusted
Mean
(Treatment)
ES
11
20
5
80.99
(n=16)
78.13
(n=16)
-.15
.6983
-6.0
11
21
2
94.25
(n=15)
98.70
(n=16)
1.11**
.0067
36.7
11
22
4
89.70
(n=17)
101.75
(n=20)
.96**
.0087
33.2
11
23
1
69.33
(n=20)
82.43
(n=22)
1.04**
.0025
35.1
11
24
2
95.01
(n=15)
95.29
(n=20)
.04
.9067
1.6
11
25
1
59.23
(n=18)
61.79
(n=19)
.14
.6948
5.6
12
26
K
87.80
(n=12)
93.55
(n=17)
.46
.2565
17.7
12
27
5
90.49
(n=13)
94.27
(n=6)
.82
.1486
29.4
12
28
5
76.75
(n=17)
85.91
(n=20)
1.21**
.0013
38.7
13
29
4
79.59
(n=22)
81.54
(n=24)
.22
.4825
8.7
14
30
K
72.31
(n=11)
79.10
(n=16)
.59
.1703
22.2
15
31
9-12
81.17
(n=19)
78.24
(n=20)
-.26
.4328
-10.6
16
32
4
85.37
(n=19)
92.50
(n=18)
.76*
.0339
27.6
16
33
1
84.63
(n=14)
95.20
(n=16)
1.70***
.0001
45.5
17
34
7
68.16
(n=22)
77.66
(n=22)
.53
.0966
20.2
17
35
6
83.98
(n=20)
88.45
(n=19)
.36
.2811
14.4
18
36
5
79.87
(n=24)
93.25
(n=24)
1.50****
.0000
43.3
19
37
4
90.34
(n=12)
88.66
(n=15)
-.17
.6810
-6.8
20
38
8
92.32
(n=18)
95.16
(n=21)
.17
.6150
6.8
21
39
2
78.48
(n=17)
70.58
(n=17)
-.72
.0540
-26.4
Final Report
14
2009/2010 Second Year Promethean Evaluation Study
Sig.
(2-tailed)
% Gain
Marzano Research Laboratory
December 2010
Site
Teacher
Grade
Adjusted
Mean
(Control)
Adjusted
Mean
(Treatment)
ES
22
40
10-12
66.69
(n=29)
78.39
(n=23)
.68*
.0215
25.2
23
41
3
72.60
(n=11)
74.02
(n=19)
.14
.7352
5.6
24
42
2
58.23
(n=10)
64.76
(n=18)
.55
.1966
20.9
25
43
--
80.33
(n=11)
81.70
(n=12)
.13
.7710
5.2
26
44
--
18.60
(n=18)
18.94
(n=15)
.10
.7905
4.0
67.14
(n=12)
*p < .05; **p < .01; ***p < .001; ****p < .0001.
65.82
(n=15)
-.06
.8762
-2.8
27
45
3
Sig.
(2-tailed)
% Gain
Figure 18 presents the findings for 46 treatment/control studies (note: one of the 45 teachers submitted
two studies). When considering the information displayed in this figure, it should be noted that the data
for each treatment/control study were checked for obvious coding errors, negative gains, and other
potential outliers. A reasonable assumption can be made that students would learn more about academic
content addressed during a unit of instruction as the unit progressed. It would not make sense for a
student to know less about academic content at the end of a unit. Based on this assumption, students who
scored higher on the pretest than the posttest were excluded from analysis. The student counts reflected
in this report represent the number of students whose test scores were considered for analysis in the
continuation study (i.e., students who took both pretest and posttest that did not score higher on the
pretest).
The columns labeled “ES” and “% Gain” are of considerable interest. Again, the column entitled “ES”
contains the standardized mean difference effect size (Cohen’s d) for each treatment/control study that
was computed from the ANCOVA findings for each study. The column entitled “% Gain” contains the
percentile gain (or loss) in achievement associated with the treatment (i.e., use of Promethean
ActivClassroom). The values in this column were determined by consulting a normal curve table for the
area for each reported effect size. A positive value in these columns indicates a finding that favors the
treatment group.
Consider the results reported for Teacher 3 in the third row of Figure 18. The adjusted mean for the
control group was 82.46 (n = 17) and the adjusted mean for the treatment group was 95.18 (n = 17). The
percentile gain associated with the effect size (d = 1.03) for this study was 34.9. This means that the
average score in the treatment group was 1.03 of a standard deviation or 34.9 percentile points greater
than the average score in the control group. In some cases the reported percentile gain was negative.
This occurs when the adjusted mean for the treatment group is less than the adjusted mean for the
control group. For example, the percentile “gain” reported for Teacher 1 in the first row was negative
23.9 (-23.9, d = -.64). This means that the average score in the treatment group was .64 of a standard
deviation or 23.9 percentile points lower than the average score in the control group. Stated differently,
Final Report
15
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
the average score in the control group was 23.9 percentile points greater than the average score in the
treatment group. (See Technical Note 7 for a more detailed explanation of percentile gain.)
In social science research and evaluation it is common practice to consider any contrast with a
probability less than .05 as “statistically significant” (see Murphy & Myors, 2004). Using this
convention, Figure 18 indicates that the comparison between treatment and control groups was
statistically significant for 15 treatment/control studies at the .05 level (p <. 05). Of those 15 studies, 12
would also be considered statistically significant at the .01 level (p < .01), 5 would be considered
statistically significant at the .001 level (p < .001), and 2 would be considered statistically significant at
the .0001 level. Consider the p-value reported for Teacher 3, p = .0072. This value is generally
interpreted to mean that only 7 times in 1,000 would sampling error alone be responsible for the
observed difference between group means. The higher this value, the more likely that sampling error
alone explains the variance between groups. Again, 15 out of the 46 studies (or 32.6%) were considered
statistically significant (p < .05). Conversely, 67.4% were not considered statistically significant. Using
meta-analytic techniques, aggregate findings might be considered statistically significant even though a
number of the individual studies were not found to be significant at the .05 level (p < .05). Such is the
case with the present set of studies.
For comparison, Figure 19 shows the overall weighted mean effect size (Cohen’s d) for a meta-analysis
of 46 treatment/control studies (continuation study) and 131 treatment/control studies (combined
continuation and first year studies) using a random-effects model of error (see Technical Note 8 for
discussion of fixed-effect vs. random-effects meta-analysis). The column labeled “N” identifies the
����” reports the weighted
number of treatment/control studies included in the group, the column labeled “𝐸𝑆
mean effect size (Cohen’s d) for the studies, the column labeled “SE” contains the standard error for the
reported weighted mean effect size, the column labeled “95% CI” identifies the 95% confidence interval
(lower limit and upper limit) for the reported weighted mean effect size, the column labeled “Sig.”
reports the p-value (2-tailed) for the reported weighted mean effect size, the column labeled “% Gain”
contains the percentile gain (or loss) associated with the reported weighted mean effect size, and the
column labeled “Fail-Safe N” identifies the number of additional studies with an effect size of .00 that
would be required to reduce the weighted mean effect size to .01 using Orwin’s formula (for a
discussion of sampling bias and the fail-safe N, see Lipsey & Wilson, 2001, pp. 165-166).
Figure 19. Overall Random Effects
Evaluation
Study
N
Continuation
46
95% CI
% Gain
Fail-Safe
N
.0014
(.0011)
13.3
(15.2)
1,518
(1,748)
.36****
.05
.25
.46
.0000
(.41)****
(.06)
(.29)
(.53)
(.0000)
Note: Corrected findings are presented in parentheses. ** p < .01; **** p < .0001.
14.1
(15.9)
4,585
(5,240)
Combined
����
𝑬𝑺
.34**
(.39)**
SE
.11
(.12)
LL
UL
.13
(.16)
.55
(.62)
131
Sig.
(2-tailed)
When the results of the 46 treatment/control studies from the continuation study were corrected for
attenuation and combined into a weighted average effect size, the corrected effect size was statistically
significant at the .01 level (d = .39, p < .01). (For a discussion of combining effect sizes and computing
significance levels see Lipsey & Wilson, 2001; for a detailed discussion of attenuation see Hunter &
Final Report
16
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Schmidt, 2004.) The overall percentile gain was 15.2. This means that on the average, the use of
Promethean ActivClassroom in the continuation study was associated with a gain in student academic
achievement of 15.2 percentile-points over what was expected when teachers did not use Promethean
ActivClassroom (see Technical Note 7 for a more thorough explanation).
Consider the fail-safe N reported in parentheses for the continuation study, 1,748. This indicates that
over 1,700 additional treatment/control studies with an effect size of .00 (i.e., no difference between
groups) would be needed to reduce the weighted mean effect size of .39 to an effect size of .01. In other
words, it would take about 1,750 treatment/control studies with no effect (d = .00) to lower the weighted
mean effect size (d = .39) to the smallest positive effect size (d = .01).
The column labeled “95% CI” contains the 95% confidence interval for the reported weighted mean
effect size. Again, the effect size reported in Figure 19 is a weighted average of all the standardized
mean difference effect sizes from the 46 treatment/control studies (see Figure 18). As such, it is
considered an estimate of the true effect size of the treatment (i.e., use of Promethean ActivClassroom).
This interval includes the range of effect sizes in which one can be 95% certain the true effect size falls.
For example, consider the confidence interval reported in parentheses for the continuation study. There
is a 95% certainty that the true effect size for the meta-analysis of the 46 treatment/control studies is
between the values of .16 and .62. When the confidence interval does not include .00, the weighted
mean effect size is determined to be statistically significant at the .05 level (p < .05). In other words, d =
.00 would not be considered a reasonable assumption. (For a detailed discussion of the meaning of
statistical significance, see Harlow, Muliak, & Steiger, 1997.)
These findings are consistent with those reported for the first year evaluation study (see Marzano &
Haystead, 2009). When the results of the first year evaluation study and continuation study were
corrected for attenuation and combined, the overall percentile gain was 15.9 (d = .41, N = 131, p <
.0001).
It is common practice when aggregating effect sizes through meta-analysis to perform a test to
determine whether those effect sizes are homogeneous. Figure 20 reports the results of the homogeneity
analysis for the weighted mean effect sizes listed in Figure 19. In basic terms, homogeneity analysis
compares the observed variance in effect sizes to the variance that would be expected from sampling
error alone (see Technical Note 9 for a discussion of homogeneity analysis).
Figure 20. Homogeneity Analysis
Evaluation
Study
Q
Continuation
177.704****
(223.659)****
Sig.
(2-tailed)
.0000
(.0000)
df
45
400.262****
.0000
130
(511.723)****
(.0000)
Note: Corrected findings are presented in parentheses. **** p < .0001
Combined
Final Report
17
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
All of the Q-values were statistically significant (p < .0001). Therefore, a reasonable assumption can be
made that the observed variances in effect sizes is greater than what would likely result from sampling
error alone. This suggests that further examination of study characteristics is warranted to determine
whether any characteristic is systematically associated with the observed variances.
Another way to examine the general effect of the use of Promethean ActivClassroom in the classroom is
to consider the distribution of effect sizes. Figure 21 presents the distribution of effect sizes for the
continuation study. Figure 22 depicts the distribution of effect sizes for the combined continuation and
first year studies.
Figure 21. Distribution of Effect Sizes – Continuation Study
Figure 21 displays the distribution of “groups” of effect sizes across the 46 treatment/control studies
from the continuation study reported in Figure 18 (M = .33, SD = .81, Min = -1.70, Max = 2.54). 3
studies exhibited an effect size less than -1.00 (see first three columns), 9 studies exhibited an effect size
between -1.00 and .00 (see fourth through sixth columns), 26 studies exhibited an effect size between
.00 and 1.00 (see seventh through ninth columns), and so on. 34 out of the 46 studies (or 73.9%) had a
positive effect size. This is very close to the percentage reported (78%) in the first year evaluation study
(Marzano & Haystead, 2009).
Final Report
18
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 22. Distribution of Effect Sizes – Combined Continuation and First Year Studies
Figure 22 displays the distribution of effect sizes across the 131 treatment/control studies from the
combined continuation and first year studies (M = .36, SD = .65, Min = -1.70, Max = 2.54). 31 studies
exhibited a negative effect (see first through sixth columns), 80 studies exhibited an effect size between
.00 and 1.00 (see seventh through tenth columns), 19 studies exhibited an effect size between 1.00 and
2.00 (see eleventh through fourteenth columns), and 1 study exhibited an effect size greater than 2.00
(see last column). 100 out of the 131 studies (or 76.3%) had a positive effect size.
The following sections present the findings from a random-effects meta-analysis for each question
involving a categorical variable. Each variable was treated as a moderator in the meta-analysis. A
moderator is a qualitative or quantitative variable that affects the direction and/or strength of the relation
between the dependent and independent variables. As Figures 21 and 22 demonstrate, there was a
substantial variation in effect sizes between treatment/control studies. A moderator analysis can help
offer insight as to why effect sizes vary from one study to another. The findings will be reported in the
same manner as for the overall random effects.
Final Report
19
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 2: Does the effect of Promethean ActivClassroom differ between school
levels?
In order to answer this question, a random-effects meta-analysis was employed using the school level for
each treatment/control study as a moderator variable. The meta-analytic findings for this variable are
reported in Figures 23 and 24.
Figure 23. Random Effects – School Level
School Level
Elementary School
(Grades K-5)
Middle School
(Grades 6-8)
High School
(Grades 9-12)
Evaluation
Study
N
Continuation
30
.27*
(.31)*
Combined
66
Continuation
95% CI
����
𝑬𝑺
SE
Sig.
(2-tailed)
% Gain
LL
UL
.13
(.15)
.01
(.02)
.52
(.60)
.0389
(.0348)
10.6
(12.2)
.38****
(.43)****
.08
(.09)
.23
(.27)
.52
(.60)
.0000
(.0000)
14.8
(16.6)
11
.71***
(.82)***
.21
(.24)
.29
(.34)
1.13
(1.30)
.0009
(.0007)
26.1
(29.4)
Combined
43
.33***
(.38)***
.09
(.10)
.15
(.18)
.50
(.58)
.0003
(.0002)
12.9
(14.8)
Continuation
5
-.07
(-.10)
.33
(.38)
-.72
(-.84)
.58
(.63)
.8232
(.7831)
-2.8
(-4.0)
.36**
.13
.10
.61
.0069
14.1
(.40)**
(.15)
(.11)
(.69)
(.0074)
(15.5)
Note: See discussion of Figure 19 for a description of column headings. Corrected findings are presented in parentheses.
* p < .05; ** p < .01; *** p < .001; **** p < .0001.
Combined
22
Figure 24. Homogeneity Analysis – School Level
Evaluation
Study
Qb
Continuation
4.839
(5.176)
Sig.
(2-tailed)
.0890
(.0752)
.184
.9119
(.166)
(.9204)
Note: Corrected findings are presented in parentheses.
Combined
df
2
2
Figure 23 depicts the estimates of effect size from a random-effects meta-analysis for the elementary,
middle, and high school levels. The weighted mean effect sizes reported for the continuation study were
statistically significant for middle school (d = .82, N = 11, p < .001) and elementary school (d = .31, N =
30, p < .05). The weighted mean effect sizes reported for the combined continuation and first year
studies were statistically significant for elementary school (d = .43, N = 66, p < .0001), middle school (d
= .38, N = 43, p < .001), and high school (d = .40, N = 22, p < .01). The effect sizes were positive for
elementary and middle school in the continuation study. When combined with the first year evaluation
study, the effect sizes were positive for all three school levels.
Final Report
20
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 24 reports the results of the homogeneity analysis for the levels of this moderator variable—
school level (see Technical Note 9 for a discussion of homogeneity analysis and moderator variables).
Neither the Q-value for the continuation or combined study was statistically significant (p < .05). A
reasonable assumption can be made that school level was not a significant contributor to the variation in
effect sizes.
Evaluation Question 3: Does the effect of Promethean ActivClassroom differ between grade
levels?
In order to answer this question, a random-effects meta-analysis was employed using the reported grade
level for each treatment/control study as a moderator variable. The meta-analytic findings for this
variable are reported in Figures 25 and 26.
Figure 25. Random Effects – Grade Level
Grade
Evaluation
Study
N
Continuation
2
.52
(.61)
Combined
3
Continuation
����
𝑬𝑺
95% CI
Sig.
SE
(2-tailed)
% Gain
LL
UL
.55
(.62)
-.55
(-.61)
1.60
(1.82)
.3386
(.3276)
19.9
(22.9)
.42
(.48)
.39
(.43)
-.34
(-.36)
1.78
(1.32)
.2763
(.2652)
16.3
(18.4)
6
.15
(.17)
.31
(.35)
-.46
(-.51)
.75
(.86)
.6334
(.6190)
6.0
(6.8)
Combined
11
.33
(.39)
.19
(.21)
-.03
(-.02)
.70
(.80)
.0750
(.0678)
12.9
(15.2)
Continuation
4
.23
(.27)
.38
(.43)
-.52
(-.58)
.98
(1.12)
.5440
(.5333)
9.1
(10.6)
Combined
9
.37
(.43)
.21
(.24)
-.04
(-.04)
.78
(.89)
.0802
(.0726)
14.4
(16.6)
Continuation
5
.16
(.18)
.34
(.39)
-.51
(-.57)
.83
(.94)
.6382
(.6339)
6.4
(7.1)
Combined
10
.35
(.40)
.20
(.22)
-.04
(-.04)
.74
(.84)
.0757
(.0751)
13.7
(15.5)
Continuation
7
.05
(.06)
.28
(.32)
-.50
(-.57)
.61
(.70)
.8484
(.8497)
2.0
(2.4)
Combined
9
.03
(.04)
.21
(.23)
-.37
(-.42)
.44
(.49)
.8692
(.8755)
1.2
(1.6)
Continuation
5
.80*
(.92)*
.35
(.39)
.13
(.16)
1.48
(1.69)
.0198
(.0181)
28.8
(32.1)
Combined
18
.54***
(.62)***
.14
(.16)
.26
(.30)
.82
(.94)
.0002
(.0001)
20.5
(23.2)
Continuation
1
.37
(.42)
.74
(.85)
-1.09
(-1.24)
1.82
(2.08)
.6228
(.6186)
14.4
(16.3)
Combined
17
.37*
(.43)*
.15
(.17)
.08
(.10)
.66
(.76)
.0127
(.0116)
14.4
(16.6)
K
1
2
3
4
5
6
Final Report
21
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Grade
Evaluation
Study
N
Continuation
4
.70
(.81)
Combined
14
Continuation
95% CI
����
𝑬𝑺
Sig.
SE
(2-tailed)
% Gain
LL
UL
.38
(.43)
-.05
(-.04)
1.44
(1.65)
.0656
(.0614)
25.8
(29.1)
.19
(.22)
.16
(.18)
-.12
(-.14)
.50
(.58)
.2351
(.2227)
7.5
(8.7)
6
.79*
(.92)*
.32
(.36)
.17
(.22)
1.41
(1.62)
.0122
(.0104)
28.5
(32.1)
Combined
12
.44*
(.52)*
.18
(.20)
.09
(.12)
.79
(.91)
.0133
(.0105)
17.0
(19.9)
Continuation
1
-.29
(-.33)
.74
(.84)
-1.73
(-1.98)
1.16
(1.32)
.6949
(.6921)
-11.4
(-12.9)
Combined
7
.33
(.39)
.24
(.27)
-.13
(-.14)
.80
(.91)
.1568
(.1462)
12.9
(15.2)
Continuation
--
--
--
--
--
--
--
Combined
--
--
--
--
--
--
--
Continuation
--
--
--
--
--
--
--
Combined
--
--
--
--
--
--
--
Continuation
--
--
--
--
--
--
--
7
8
9
10
11
12
.13
.69
-1.22
1.49
.8485
5.2
(.15)
(.77)
(-1.35)
(1.65)
(.8425)
(6.0)
Note: See discussion of Figure 19 for a description of column headings. Corrected findings are presented in parentheses.
* p < .05; *** p < .001.
Combined
1
Figure 26. Homogeneity Analysis – Grade Level
Evaluation
Study
Qb
(2-tailed)
Continuation
7.248
(7.485)
.6113
(.5687)
Sig.
5.499
.8554
(5.711)
(.8389)
Note: Corrected findings are presented in parentheses.
Combined
df
9
10
Figure 25 displays the random effects for each grade level, K-12. In the continuation study, two studies
did not report a grade level and three studies reported more than one grade level. Therefore, five
treatment/control studies were excluded from the grade level analysis. The weighted mean effect sizes
reported in Figure 25 were statistically significant for two grade levels in the continuation study: fifth
grade (d = .92, N = 5, p < .05) and eighth grades (d = .92, N = 6, p < .05). The effect sizes were positive
for nine grade levels. When combined with the findings from the first year evaluation study, the
Final Report
22
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
weighted mean effect sizes were statistically significant for fifth grade (d = .62, N = 18, p < .001), sixth
grade (d = .43, N = 17, p < .05), and eighth grade (d = .52, N = 12, p < .05). The effect sizes were
positive for eleven grade levels. Altogether, 20 treatment/control studies were excluded from the
combined analysis.
Figure 26 depicts the results of the homogeneity analysis for the levels of this moderator variable—
grade level. For this moderator variable, the between studies Q-values were not statistically significant
(p < .05). Therefore, a reasonable assumption can be made that grade level was not a significant
contributor to the variation in effect sizes.
Evaluation Question 4: Does the effect of Promethean ActivClassroom differ between academic
content areas?
In order to answer this question, a random-effects meta-analysis was employed using the reported
academic content area (see Appendix C) for each treatment/control study as a moderator variable. The
meta-analytic findings for this variable are reported in Figures 27 and 28.
Figure 27. Random Effects – Academic Content Area
School Level
Evaluation
Study
N
Continuation
95% CI
����
𝑬𝑺
SE
13
.33
(.38)
Combined
33
Continuation
Sig.
(2-tailed)
% Gain
LL
UL
.20
(.23)
-.07
(-.07)
.72
(.82)
.1029
(.0959)
12.9
(14.8)
.35***
(.40)***
.11
(.12)
.14
(.17)
.55
(.64)
.0009
(.0007)
13.7
(15.5)
18
.23
(.27)
.17
(.19)
-.09
(-.10)
.56
(.64)
.1606
(.1533)
9.1
(10.6)
Combined
48
.31***
(.36)***
.09
(.10)
.14
(.17)
.48
(.55)
.0003
(.0003)
12.2
(14.1)
Continuation
9
.77**
(.90)**
.24
(.28)
.30
(.36)
1.25
(1.44)
.0014
(.0011)
27.9
(31.6)
Combined
24
.52***
(.60)***
.12
(.14)
.28
(.33)
.76
(.87)
.0000
(.0000)
19.9
(22.6)
Continuation
4
.26
(.30)
.35
(.40)
-.44
(-.49)
.95
(1.09)
.4657
(.4543)
10.3
(11.8)
Language Arts
Mathematics
Science
Social Studies
.24
.14
-.02
.51
.0731
9.5
(.28)
(.15)
(-.02)
(.58)
(.0655)
(11.0)
Note: See discussion of Figure 19 for a description of column headings. Corrected findings are presented in parentheses.
** p < .01; *** p < .001.
Combined
19
Final Report
23
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 28. Homogeneity Analysis – Academic Content Area
Evaluation
Study
Qb
Continuation
3.611
(3.764)
Sig.
(2-tailed)
.3066
(.2881)
2.797
.4239
(2.876)
(.4111)
Note: Corrected findings are presented in parentheses.
Combined
df
3
3
Figure 27 lists the meta-analytic findings for the treatment/control studies that involved four academic
content areas: language arts, mathematics, science, and social studies. In the continuation study, two
studies involved unrelated content areas and were excluded from this meta-analysis. For the continuation
study, the weighted mean effect size reported for science was statistically significant (d = .90, N = 9, p <
.01). The effect sizes were positive for all four academic content areas. When the findings were
combined with the first year evaluation study, the weighted mean effect sizes were statistically
significant for language arts (d = .40, N = 33, p < .001), mathematics (d = .36, N = 48, p < .001), and
science (d = .60, N = 24, p < .001). The effect sizes were positive for all content areas. Seven
treatment/control studies involved unrelated content areas and were excluded from the combined
analysis.
Figure 28 displays the results of the homogeneity analysis for the levels of this moderator variable—
academic content area. For this moderator variable, the between studies Q-values were not statistically
significant (p < .05). Therefore, a reasonable assumption can be made that academic content area was
not a significant contributor to the variation in effect sizes.
Evaluation Question 5: Is there a relationship between the effect of Promethean ActivClassroom
and length of teaching experience, how long teachers have used Promethean ActivClassroom, the
percentage of time Promethean ActivClassroom was used in the classroom, and teachers’ selfreported confidence in their use of Promethean ActivClassroom?
In order to answer this question, SPSS was used to compute the product moment correlation coefficient
(Pearson’s r) for a comparison between the observed effect sizes (Cohen’s d) from Figure 18 and the
four teacher survey responses (see Appendix C): length of teaching experience (reported in years), how
long Promethean ActivClassroom has been used (reported in months), percentage of time Promethean
ActivClassroom was used in the classroom, and teachers’ confidence in their ability to use Promethean
ActivClassroom. For comparison purposes, Figures 29 through 36 display the distribution for each of the
survey responses.
Final Report
24
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 29. Distribution of Teaching Experience – Continuation Study
Figure 29 shows that for the continuation study, 21 of the treatment/control studies were conducted by
teachers with less than 10 years of experience, 17 studies were conducted by teachers with between 10
and 20 years of experience, 6 studies were conducted by teachers with between 20 and 30 years of
experience, and 1 study was conducted by a teacher with more than 30 years of experience. (Note:
survey data missing for one study.)
Final Report
25
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 30. Distribution of Teaching Experience – Combined Continuation and First Year Studies
Figure 30 indicates that for the combined continuation and first year studies, 55 of the treatment/control
studies were conducted by teachers with less than 10 years of experience, 44 studies were conducted by
teachers with between 10 and 20 years of experience, 14 studies were conducted by teachers with
between 20 and 30 years of experience, and 3 studies were conducted by teachers with more than 30
years of experience. (Note: survey data missing for 15 studies.)
Final Report
26
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 31. Distribution of Promethean ActivClassroom Use – Continuation Study
Figure 31 shows that for the continuation study, 15 of the treatment/control studies were conducted by
teachers who reported having used Promethean ActivClassroom for less than 10 months, 14 studies were
conducted by teachers who reported having used Promethean ActivClassroom between 10 and 20
months, 4 studies were conducted by teachers who reported having used Promethean ActivClassroom
between 20 and 30 months, 9 studies were conducted by teachers who reported having used Promethean
ActivClassroom between 30 and 40 months, and 2 studies were conducted by teachers who reported
having used Promethean ActivClassroom for more than 40 months. (Note: survey data missing for two
studies.)
Final Report
27
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 32. Distribution of Promethean ActivClassroom Use – Combined Continuation and First
Year Studies
Figure 32 indicates that for the combined continuation and first year studies, 38 of the treatment/control
studies were conducted by teachers who reported having used Promethean ActivClassroom for less than
10 months, 28 studies were conducted by teachers who reported having used Promethean
ActivClassroom between 10 and 20 months, 20 studies were conducted by teachers who reported having
used Promethean ActivClassroom between 20 and 30 months, 21 studies were conducted by teachers
who reported having used Promethean ActivClassroom between 30 and 40 months, and 8 studies were
conducted by teachers who reported having used Promethean ActivClassroom for more than 40 months.
(Note: survey data missing for 16 studies.)
Final Report
28
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 33. Distribution of Percentage of Class Time – Continuation Study
Figure 33 indicates that for the continuation study, 4 of the treatment/control studies were conducted by
teachers who reported having used Promethean ActivClassroom between 20 and 40 percent of their
classroom instructional time, 12 studies were conducted by teachers who reported having used
Promethean ActivClassroom between 40 and 60 percent of their classroom instructional time, 8 studies
were conducted by teachers who reported having used Promethean ActivClassroom between 60 and 80
percent of their classroom instructional time, and 19 studies were conducted by teachers who reported
having used Promethean ActivClassroom between 80 and 100 percent of their classroom instructional
time. (Note: survey data missing for three studies.)
Final Report
29
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 34. Distribution of Percentage of Class Time – Combined Continuation and First Year
Studies
Figure 34 shows that for the combined continuation and first year studies, one treatment/control study
was conducted by a teacher who reported having used Promethean ActivClassroom less than 20 percent
of his/her classroom instructional time, 15 studies were conducted by teachers who reported having used
Promethean ActivClassroom between 20 and 40 percent of their classroom instructional time, 24 studies
were conducted by teachers who reported having used Promethean ActivClassroom between 40 and 60
percent of their classroom instructional time, 28 studies were conducted by teachers who reported
having used Promethean ActivClassroom between 60 and 80 percent of their classroom instructional
time, and 46 studies were conducted by teachers who reported having used Promethean ActivClassroom
between 80 and 100 percent of their classroom instructional time. (Note: survey data missing for 17
studies.)
Final Report
30
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 35. Distribution of Confidence Ratings – Continuation Study
Final Report
31
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 36. Distribution of Confidence Ratings – Combined Continuation and First Year Studies
Figure 35 shows that a majority of the teachers in the continuation study rated themselves as mostly
confident in their ability to use Promethean ActivClassroom. (Note: survey data missing for one study.)
Figure 36 indicates a similar majority for the combined continuation and first year studies. (Note: survey
data missing for 15 studies.)
Figures 29 through 36 provide a graphical depiction of the four survey responses. Figure 37 lists the
product moment correlation coefficients (Pearson’s r) between the observed effect sizes (Cohen’s d)
from the continuation study reported in Figure 18 and the four survey responses. Figure 38 displays the
correlation coefficients (Pearson’s r) between the observed effect sizes (Cohen’s d) from the combined
continuation and first year studies and the four survey responses.
Final Report
32
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 37. Product Moment Correlations for Teacher Survey Responses – Continuation Study
Comparison
Correlation
Effect Size
Years of
Teaching
Experience
Months of
Technology
Use
Percentage of
Class Time
Confidence
Effect Size
# Yrs Teach
#Mos Tech
% Class
Time
Confidence
1
-.09
-.21
.22
-.08
.545
.168
.155
.581
Sig. (2-tailed)
N
46
45
44
43
45
Correlation
-.09
1
.15
-.18
-.02
Sig. (2-tailed)
.545
.347
.248
.912
N
45
45
44
43
45
Correlation
-.21
.15
1
.13
.57***
Sig. (2-tailed)
.168
.347
.409
.000
N
44
44
44
42
44
Correlation
.22
-.18
.13
1
.19
Sig. (2-tailed)
.155
.248
.409
N
43
43
42
43
43
Correlation
-.08
-.02
.57***
.19
1
Sig. (2-tailed)
.581
.912
.000
.222
N
45
45
44
43
.222
45
*** p < .001.
Figure 37 shows that no statistically significant correlations (p < .05) were found between the observed
effect sizes and the teacher survey responses. However, the correlation between the number of months
Promethean ActivClassroom was used and teachers’ confidence in their ability to use Promethean
ActivClassroom was positive and significant (r = .57, N = 44, p < .001).
Final Report
33
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 38. Product Moment Correlations for Teacher Survey Responses – Combined
Continuation and First Year Studies
Comparison
Correlation
Effect Size
Years of
Teaching
Experience
Months of
Technology
Use
Percentage of
Class Time
Confidence
Effect Size
Yrs Teach
Mos Tech
% Class
Time
Confidence
1
-.07
-.03
.22*
.04
.490
.725
.022
.693
Sig. (2-tailed)
N
131
116
115
114
116
Correlation
-.07
1
.01
-.18
-.21*
Sig. (2-tailed)
.490
.910
.064
.026
N
116
116
115
113
116
Correlation
-.03
.01
1
.11
.50***
Sig. (2-tailed)
.725
.910
.272
.000
N
115
115
115
112
115
Correlation
.22*
-.18
.11
1
.45***
Sig. (2-tailed)
.022
.064
.272
N
114
113
112
114
113
Correlation
.04
-.21*
.50***
.45***
1
Sig. (2-tailed)
.693
.026
.000
.000
N
116
116
115
113
.000
116
* p < .05; *** p < .001.
Figure 38 indicates that the correlation between the observed effect sizes (combined continuation and
first year studies) and the percentage of time Promethean ActivClassroom was used in the classroom
was positive and significant (r = .22, N = 114, p < .05). In addition, statistically significant correlations
were found between teachers’ confidence and the number of years of teaching experience (r = -.21, N =
116, p < .05), the number of months Promethean ActivClassroom was used (r = .50, N = 115, p < .001),
and the percentage of time Promethean ActivClassroom was used (r = .45, N = 113, p < .001).
Final Report
34
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Phase I Summary and Interpretation
The results of the continuation study are rather straight-forward and might be summarized by the
following statements:
•
•
•
•
When the effect sizes from the 46 treatment/control studies were corrected for attenuation due to
unreliability of the dependent measure, the overall effect for the use of Promethean
ActivClassroom was .39 and was significant at the .01 level (p < .01).
No significant relationship was found between the 46 effect sizes associated with the use of
Promethean ActivClassroom and the following teacher survey response variables:
o length of teaching experience (reported in years)
o how long teachers have used Promethean ActivClassroom (reported in months)
o the percentage of time Promethean ActivClassroom was used in the classroom
o teachers’ self-reported confidence in their use of Promethean ActivClassroom
A significant relationship was found between teachers’ self-reported confidence in their ability to
use Promethean ActivClassroom and the number of months Promethean ActivClassroom was
used (r = .57, N = 44, p < .001).
No significant mediator influence was found for the effects of Promethean ActivClassroom for
any of the following variables: school level, grade level, and academic content area.
Perhaps even more important than the findings from the continuation study are the combined findings
from the first year and continuation studies. These combined findings can be summarized in the
following statements:
•
When the effect sizes from the 131 treatment/control studies were corrected for attenuation due to
unreliability of the dependent measure, the overall effect for the use of Promethean ActivClassroom
was .41 and was significant at the .0001 level (p < .0001).
This statement implies that the observed differences between treatment and control groups could have
occurred less than one time in ten thousand if there was no true relationship between Promethean
ActivClassroom and student academic achievement. A reasonable inference is that the 15.9 percentilepoint gain associated with the overall weighted average effect size of .41 is probably not a function of
random factors that are specific to the treatment/control studies. Stated differently, a reasonable
assumption is that the use of Promethean ActivClassroom is associated with a gain in student
achievement.
•
A significant relationship (r = .22, N = 114, p < .05) was found between the 131 effect sizes
associated with the use of Promethean ActivClassroom and the percentage of time Promethean
ActivClassroom was used in the classroom.
This statement implies that the effect of Promethean ActivClassroom on student academic achievement
increases as the percentage of classroom instructional time that Promethean ActivClassroom is utilized
increases. Marzano and Haystead (2009) reported findings that suggested substantial gains in student
achievement under the following conditions: a teacher has 10 years or more of teaching experience, a
teacher has used Promethean ActivClassroom for two years or more, a teacher uses Promethean
ActivClassroom between 75 and 80 percent of the time in his or her classroom, and a teacher has high
Final Report
35
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
confidence in his or her ability to use Promethean ActivClassroom. The correlations reported in Figure
38 indicate that except for the percentage of class time in which Promethean ActivClassroom was used,
those conditions do not have a relationship with effect size. Relative to the percentage of time
Promethean ActivClassroom was used, the positive relationship with effect size does not decrease after
80 percent of class time.
•
Significant relationships were found between teachers’ self-reported confidence in their ability to use
Promethean ActivClassroom and the following teacher survey response variables:
o length of teaching experience [reported in years] (r = -.21, N = 116, p < .05)
o how long teachers have used Promethean ActivClassroom [reported in months]
(r = .50, N = 115, p < .001)
o the percentage of time Promethean ActivClassroom was used in the classroom
(r = .45, N = 113, p < .001)
This statement implies that teachers’ confidence ratings were influenced by how long they have been a
classroom teacher and how long they have used Promethean ActivClassroom. Additionally, this
statement implies that the percentage of time Promethean ActivClassroom was used in the classroom
was influenced by teachers’ confidence ratings.
The negative correlation between confidence and teaching experience suggests that as the length of
teaching experience increased confidence ratings decreased. In other words, some of the teachers in the
evaluation study with several years of experience in the classroom were less confident in their ability to
use Promethean ActivClassroom. Conversely, some of the teachers with a few years of experience in the
classroom were more confident in their ability to use Promethean ActivClassroom. Again, 99 of the 131
effect sizes (or 75.6%) were contributed by teachers with less than 20 years of teaching experience (see
Figure 30). The most parsimonious explanation for the negative relationship between confidence and the
number of years of teaching experience might simply be the lack of effect sizes contributed by teachers
with more than 20 years of teaching experience. Should additional treatment/control studies be
conducted by teachers with more than 20 years of experience, the finding reported here would most
likely change.
The positive correlation between confidence and the number of months Promethean ActivClassroom
was used suggests that as the use of Promethean ActivClassroom increased confidence ratings increased.
Stated differently, teachers in the evaluation study who had used Promethean ActivClassroom for the
longest period of time were more confident in their ability to use Promethean ActivClassroom. A
reasonable inference can be made that increased confidence will result from increased use of
Promethean ActivClassroom over a period of time.
The positive correlation between confidence and the percentage of time Promethean ActivClassroom
was used in the classroom suggests that the amount of instructional time Promethean ActivClassroom
was used increased as confidence ratings increased. In other words, teachers in the evaluation study with
higher confidence ratings used Promethean ActivClassroom more often during instructional time. This
implies that increased use of Promethean ActivClassroom will result from increased confidence in the
ability to use Promethean ActivClassroom. Additionally, given that there was a positive correlation
between the percentage of time Promethean ActivClassroom was used in the classroom and effect size, a
reasonable inference can be made that teachers’ confidence has an indirect relationship with effect size.
Final Report
36
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
•
No significant mediator influence was found for the effects of Promethean ActivClassroom for any
of the following variables: school level, grade level, and academic content area.
This statement implies that school level, grade level, and academic content area were not significant
contributors to the variance between the 131 effect sizes. Some other aspect (or aspects) of the
treatment/control studies may have contributed to that variance. The findings for these three variables
suggest that Promethean ActivClassroom can be equally effective at all school levels, grade levels, and
academic content areas. Marzano and Haystead (2009) reported lower than anticipated effects at seventh
grade. The random effects reported in Figure 25 indicate that additional studies did improve the average
effect for this grade level. Given the significant findings from the homogeneity analysis reported in
Figure 20 the most likely explanation for the variance in effect sizes would be characteristics associated
with the classroom teacher not already considered in this report.
Final Report
37
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Phase II
Final Report
38
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Phase II involved an analysis of student learning with and without Promethean ActivClassroom from the
perspective of teacher behavior as evidenced in videotapes of teachers using Promethean
ActivClassroom tools in their classrooms. Teachers in the study were asked to be videotaped for at least
one lesson in which they used Promethean ActivClassroom (i.e., treatment group) and one lesson in
which they did not (i.e., control group). This allowed for the examination of a number of evaluation
questions that were not available without the videotapes. The general focus of Phase II was to determine
the behaviors that differentiated those teachers who obtained positive effects from Promethean
ActivClassroom from those who did not. Recall that in the Phase I analysis 74% of the treatment/control
studies in the second year evaluation study exhibited greater achievement with Promethean
ActivClassroom and 26% of the treatment/control studies exhibited greater achievement without the
technology.
The evaluation questions that relate to this phase of the study were:
Evaluation Question 1: What is the relationship between student engagement and positive versus
negative effects for the use of Promethean ActivClassroom?
Evaluation Question 2: What is the relationship between teacher IWB skills and positive versus negative
effects for the use of Promethean ActivClassroom?
Evaluation Question 3: What is the relationship between student IWB skills and positive versus negative
effects for the use of Promethean ActivClassroom?
Evaluation Question 4: What is the relationship between multiple student use of the IWB and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 5: What is the relationship between student independent use of the IWB and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 6: What is the relationship between teacher use of IWB reinforcers and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 7: What is the relationship between teacher use of learner response systems and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 8: What is the relationship between teacher use of the IWB to represent knowledge
graphically or nonlinguistically and positive versus negative effects for the use
of Promethean ActivClassroom?
Evaluation Question 9: What is the relationship between teacher previewing of content and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 10: What is the relationship between teacher chunking of content and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 11: What is the relationship between teacher scaffolding of content and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 12: What is the relationship between teacher pacing of content and positive versus
negative effects for the use of Promethean ActivClassroom?
Evaluation Question 13: What is the relationship between teacher monitoring of student progress and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 14: What is the relationship between clarity of content as depicted in the IWB and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 15: What is the relationship between student interaction about the content and
positive versus negative effects for the use of Promethean ActivClassroom?
Final Report
39
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 16: What is the relationship between student response rates and positive versus
negative effects for the use of Promethean ActivClassroom?
Evaluation Question 17: What is the relationship between classroom management and positive versus
negative effects for the use of Promethean ActivClassroom?
In general, two types of analyses were performed for each of these questions. One analysis broke the
teachers into two categories: those with positive effects and those with negative effects for the use of
Promethean ActivClassroom. Membership in these groups was considered the independent variable and
scores on the variables for the 17 Phase II evaluation questions were considered the dependent variables.
SPSS was used to perform an independent-samples t-test comparing the mean for the group with
positive effects and the mean for the group with negative effects for each of the 17 Phase II variables.
SPSS generates output for two t-tests based on assumptions of homogeneous variance: equal variances
assumed and equal variances not assumed. Unless otherwise noted, the t-value reported for each variable
was based on the t-test with equal variances assumed. (See Technical Note 10 for a discussion of the ttest and methods for assessing homogeneous variance.) For the second analysis SPSS was used to
determine the bivariate correlation (Pearson’s r) between the scores on the 17 Phase II variables and the
corrected effect sizes computed in Phase I. (See Technical Note 4 for the method that was utilized to
correct for attenuation in the observed effect sizes reported in Figure 18.)
Coding of Videotapes
Each videotape was coded by a single rater. Intra-rater reliability was computed by coding 10 videos on
all 17 variables and then re-coding those videos two weeks later without access to previous ratings.
Agreement between the two ratings was determined by analyzing the percentage of scores from the
second rating that were: 1) the same as the first rating and 2) one point (plus or minus) from the first
rating. These percentages are reported in Figure 39.
Figure 39. Percentage of Second Ratings the Same as First Rating and One Score Point from First
Rating
Variable
Same Rating
One Point Difference
Combined
1
60%
20%
80%
2
70%
20%
90%
3
80%
10%
90%
4
60%
10%
70%
5
60%
20%
80%
6
60%
10%
70%
7
70%
10%
80%
8
60%
20%
80%
9
70%
20%
90%
Final Report
40
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Variable
Same Rating
One Point Difference
Combined
10
60%
20%
80%
11
60%
40%
100%
12
60%
20%
80%
13
70%
20%
90%
14
60%
20%
80%
15
70%
20%
90%
16
60%
30%
90%
17
60%
40%
100%
As indicated in the last column of Figure 39, on 15 out of 17 Phase II variables the second rating was
either identical to the first or different by one point only (plus or minus) 80% of the time or more.
Evaluation Question 1: What is the relationship between student engagement and positive versus
negative effects for the use of Promethean ActivClassroom?
Engagement refers to the extent to which students were paying attention to the classroom activities.
High scores on this variable are exhibited by students consistently attending to the activities orchestrated
by the teacher. Figure 40 reports the results for the comparison of means for student engagement
between those teachers who exhibited positive effects for the use of Promethean ActivClassroom and
those students who exhibited negative effects for the technology.
Figure 40. Comparison of Means for Student Engagement
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
3
7
7
2
1
0
0
5.45
(N=20)
.23
Negative
1
0
3
4
0
0
0
4.75
(N=8)
.37
Positive
26
16
19
14
2
0
0
5.65
(N=77)
.14
0
4.71
(N=24)
.22
Continuation
Combined
Negative
2
4
4
13
1
0
t
Sig.
(2-tailed)
df
1.60
.122
26
3.43**
.001
99
** p < .01.
Final Report
41
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
As shown in Figure 40, the t-test reveals that in the continuation study, the positive-effects group
exhibited a higher mean score for engagement than the negative-effects group. The difference between
group means was not statistically significant at the .05 level (p < .05). When combined with the findings
from the first year evaluation study, the t-test indicates a significant difference (p < .01) between mean
scores favoring the positive-effects group.
One thing to note regarding these findings is that both mean scores were above the middle value on the
seven-point scale that was used (highest score = 7; lowest score = 1). Taken at face value, the mean
scores suggest a high central tendency for both groups on that scale. However, mean scores can
sometimes obscure characteristics of data points within a sample, especially when the sample size is
small. For example, a few data points at the high end of a scale can inflate the mean score for that data
set even though a majority of the data points are at the lower end of the scale. Therefore, it is often
useful to compare a distribution of frequencies to determine whether the true central tendency lies at,
above, or below the mean score for the distribution.
Consider the findings reported for the continuation study. In the positive-effects group, the median (i.e.,
middle value of the data set) was 5.5 and there were two modes (i.e., most frequent value of the data
set), 5 and 6. This can be interpreted to mean that the central tendency for engagement was higher than
the mean score of 5.45 for this group. Therefore, the central tendency for the positive-effects group was
very high. In the negative-effects group, the median was 4.5 and the mode was 4. This suggests that the
central tendency for engagement was lower than the mean score of 4.75 for this group. Therefore, the
central tendency for the negative-effects group was closer to the middle value on the seven-point scale.
When combined with the findings from the first year evaluation study, the median in the positive-effects
group was 6 and the mode was 7. Again, the measures of central tendency were very high for this group.
In the negative-effects group, the median and mode were both 4. This would seem to indicate a central
tendency toward the middle of the scale for this group. Taken at face value, these findings suggest that
student engagement was very high in the positive-effects group. In the negative-effects group, student
engagement appears to be more moderate.
Figure 41 reports the bivariate correlation between student engagement and the corrected effect sizes.
Technical Note 11 explains why only the corrected effect sizes were used when computing bivariate
correlations for Evaluation Questions 1 through 17 of Phase II.
Figure 41. Bivariate Correlation for Student Engagement
Pairs
Engagement &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.34
(N=28)
.079
Combined
.44***
(N=101)
.000
(2-tailed)
*** p < .001.
Final Report
42
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
In the continuation study, the bivariate correlation between student engagement and the corrected effect
sizes was positive. While the correlation was not significant at the .05 level (p < .05), the p-value was
less than .10. Because the correlation approached significance, a reasonable inference can be made that a
moderate positive linear relationship was found between effect size and student engagement. This
inference is strengthened by the significant correlation (p < .001) reported for the combined continuation
and first year studies.
Evaluation Question 2: What is the relationship between teacher IWB skills and positive versus
negative effects for the use of Promethean ActivClassroom?
Teacher IWB skills refers to the teacher’s use of features on the IWB. High scores on this variable are
exhibited by teachers utilizing various Promethean ActivClassroom tools with ease and fluency. Figure
42 reports the comparison of mean scores on this variable for the positive- and negative-effects classes.
Figure 42. Comparison of Means for Teacher IWB Skills
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
3
5
6
2
0
1
0
5.35
(N=17)
.31
Negative
0
1
2
2
1
1
0
4.14
(N=7)
.51
Positive
24
8
14
18
1
9
0
5.12
(N=74)
.20
2
4.09
(N=23)
.37
Continuation
Combined
Negative
3
2
3
8
2
3
t
Sig.
(2-tailed)
df
2.09*
.049
22
2.54*
.013
95
* p < .05.
The t-test reported in Figure 42 shows that the positive-effects group had a higher mean score with
respect to teacher IWB skills in the continuation study. The difference between group means was
statistically significant at the .05 level (p < .05). When the findings were combined with the first year
evaluation study, a significant difference (p < .05) between means was found that favors the positiveeffects group.
In the continuation study, both group means were above the middle value on the scale. In the positiveeffects group, the median and mode were both 5. In the negative-effects group, the data set had two
modes (4 and 5) with a median of 4. For both groups, the median values were lower than their respective
means.
In the combined continuation and first year studies, the median in the positive-effects group was 5 and
the mode was 7. In the negative-effects group, the median and mode were 4. This suggests a high central
tendency for the positive-effects group and a medium central tendency for the negative-effects group. In
other words, teachers in the positive-effects group appeared to utilize Promethean ActivClassroom tools
with ease and fluency more often than teachers in the negative-effects group.
Final Report
43
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 43 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 43. Bivariate Correlation for Teacher IWB Skills
Pairs
Teacher IWB Skills &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.59**
(N=24)
.003
Combined
.38***
(N=97)
.000
(2-tailed)
** p < .01; *** p < .001.
Figure 43 reveals a significant correlation (p < .01) between teacher IWB skills and effect size in the
continuation study. This finding suggests that a strong positive linear relationship was found between
effect size and teachers’ ability to utilize various Promethean ActivClassroom tools with ease and
fluency. When combined with the first year evaluation study, the correlation was reduced from .59 to .38
and was significant at the .001 level (p < .001). The combined data would seem to suggest a moderate
positive linear relationship between effect size and teachers’ skills with Promethean ActivClassroom
tools; as one increases so does the other.
Evaluation Question 3: What is the relationship between student IWB skills and positive versus
negative effects for the use of Promethean ActivClassroom?
Student IWB skills refers to the student’s ability to use features of the IWB without direct prompting
from the teacher. High scores on this variable manifest as students using features such as highlighting or
erasing without need of teacher direction. Figure 44 reports the mean scores on this variable for the
positive- and negative-effects classes.
Figure 44. Comparison of Means for Student IWB Skills
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Positive
Mean
SE
7
6
5
4
3
2
1
1
1
2
5
0
1
0
4.50
(N=10)
.67
Continuation
t
df
.43
Negative
0
0
0
2
0
1
0
3.33
(N=3)
Positive
3
1
6
28
4
7
18
3.18
(N=67)
.21
Negative
0
1
3
8
1
1
5
3.32
(N=19)
.38
Combined
Sig.
(2-tailed)
1.34
.206
11
-.32
.753
84
The t-test indicates that the positive-effects group had a higher mean score than the negative-effects
group in the continuation study. The difference in means between the two groups was not significant (p
< .05). When combined with the first year evaluation study, the negative-effects group exhibited a
higher mean score. The difference in means was not significant at the .05 level (p < .05).
Final Report
44
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
In the continuation study, the mean for the positive-effects group was above the middle value on the
scale. In the positive-effects group, the median and mode were 4. In the negative-effects group, the
median and mode were also 4. Therefore, the measures of central tendency in both groups were at the
middle of the scale for this variable.
In the combined continuation and first year studies, the means for both groups were below the middle
value on the scale. The median and mode were 4 in both groups. This suggests a central tendency at the
middle of the scale. There appears to be very little difference between groups with respect to
observations of students’ ability to use features of Promethean ActivClassroom without direct prompting
from the teacher.
Figure 45 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 45. Bivariate Correlation for Student IWB Skills
Pairs
Student IWB Skills &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.60*
(N=13)
.030
Combined
.07
(N=86)
.530
(2-tailed)
* p < .05.
Figure 45 shows that in the continuation study a significant correlation (p < .05) exists between student
IWB skills and effect size. This suggests that a strong positive linear relationship was found between
effect size and students’ ability to use features of the IWB without prompting from the teacher.
However, when combined with the first year evaluation study, the correlation was reduced from .60 to
.07. The combined finding was not significant at the .05 level (p < .05).
It should be noted that in the continuation study, the correlation was statistically significant but the
difference between means was not. This is not uncommon when a variable which is inherently ordinal,
interval, or ratio in nature is dichotomized to artificially create two groups for the purpose of comparing
differences on other variables. In this case the variable of student IWB skills is ordinal in nature with
seven possible values—1 to 7. The effect size variable is a ratio scale. In the analysis reported in Figure
44, the effect size variable was dichotomized (i.e., positive effects and negative effects) in order to make
a comparison of means on the variable of student IWB skills. This loss of information due to collapsing
of values renders the comparison between groups reported in Figure 44 less representative of the true
relationship between variables than the correlations reported in Figure 45.
Taken at face value, the findings from the combined continuation and first year studies appear to
indicate no linear relationship between effect size and student IWB skills.
Final Report
45
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 4: What is the relationship between multiple student use of the IWB and
positive versus negative effects for the use of Promethean ActivClassroom?
Multiple student use of the IWB refers to the extent that multiple students use the IWB under the
teacher’s direction. High scores on this variable manifest as many or all students in class coming to the
front of the room and using the IWB in response to directions from the teacher. Figure 46 reports the
difference in means between the positive- and negative-effects classes.
Figure 46. Comparison of Means for Multiple Student Use of the IWB
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Positive
Mean
SE
7
6
5
4
3
2
1
0
3
3
6
1
3
4
3.50
(N=20)
.39
.31
Continuation
t
Negative
0
0
0
0
2
2
4
1.75
(N=8)
Positive
8
7
8
21
6
5
22
3.53
(N=77)
.23
13
2.21
(N=24)
.32
Combined
Negative
0
2
1
3
0
5
Sig.
(2-tailed)
df
3.48**
.002
24.28
3.32**
.002
48.76
** p < .01.
In the continuation study, the difference in means between the two groups was statistically significant at
the .01 level (p < .01). It should be noted that Levene’s test for equality of variances was significant (F =
5.57, p = .026). When combined with the first year findings, the difference in means was also significant
at the .01 level (p < .01). Levene’s test was also significant (F = 4.50, p = .036). In both the continuation
and combined studies, the assumption that the two groups have approximately equal variance on the
dependent variable was not met. Therefore, the t-test for unequal variances was employed (see Technical
Note 10).
In the continuation study, both means were below the middle value on the scale. In the positive-effects
group, the median and mode were 4. In the negative-effects group, the median was 1.5 and the mode
was 1. For the positive-effects group, the median value was higher than the mean score. For the
negative-effects group, the median was lower than the mean. The measures of central tendency for this
variable were low for the negative-effects group. This indicates that observed classes exhibited very
little multiple student use of the IWB. When combined with the first year findings, both means were also
below the middle value of the scale. In the positive-effects group, the median was 4 and the mode was 1.
In the negative-effects group, the median was 2 and the mode was 1. The combined findings suggest a
low central tendency for both groups on this variable. In other words, with respect to multiple student
use of the IWB, the majority of observations were at the low end of the scale.
Figure 47 reports the bivariate correlation between this variable and the corrected effect sizes.
Final Report
46
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 47. Bivariate Correlation for Multiple Student Use of the IWB
Pairs
Multiple Student Use &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.51**
(N=28)
.006
Combined
.44***
(N=101)
.000
(2-tailed)
** p < .01; *** p < .001 .
Figure 47 indicates that a significant correlation (p < .01) exists between multiple student use of the
IWB and effect size in the continuation study. This finding would suggest an increase in effect size
when there is an increase in the utilization of the IWB by multiple students in response to teacher
directions. The combined finding reveals a significant correlation (p < .001) between multiple student
use of the IWB and effect size. A reasonable inference can be made that a strong positive linear
relationship was found between effect size and multiple student use of the IWB.
Evaluation Question 5: What is the relationship between student independent use of the IWB and
positive versus negative effects for the use of Promethean ActivClassroom?
Student independent use of the IWB refers to the extent to which students used the technology without
the direct guidance of the teacher. High scores on this variable are exhibited by students working with
the IWB in ways that are not directly guided by the teacher. Figure 48 reports the difference in means
between the positive-effects class and the negative-effects class for this variable.
Figure 48. Comparison of Means for Student Independent Use of the IWB
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
1
0
0
0
0
0
0
7.00
(N=1)
--
Negative
0
0
0
0
0
0
1
1.00
(N=1)
--
Positive
2
0
0
4
2
2
48
1.52
(N=58)
.18
13
1.47
(N=17)
.24
Continuation
Combined
Negative
0
0
0
2
0
2
t
Sig.
(2-tailed)
df
--
--
0
.13
.895
73
For this variable, there were only two treatment/control studies in the continuation study which
exhibited student independent use of the IWB, one in each group. In the positive-effects group, student
independent use was observed at the highest end of the scale. In the negative-effects group, student
independent use was observed at the lowest end of the scale. Therefore, no comparison of means was
possible with respect to student independent use of the IWB. When combined with the findings from the
first year evaluation study, the mean score was slightly higher in the positive-effects group. The
difference between means was not significant at the .05 level (p < .05).
Final Report
47
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
The combined findings reveal that both means were below the middle value of the scale. In the positiveeffects group, the median and mode were both 1. The median and mode were the same in the negativeeffects group. The combined findings suggest a very low central tendency for both groups on this
variable. In other words, the vast majority of observations of student independent use of Promethean
ActivClassroom were at the lower end of the scale.
Figure 49 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 49. Bivariate Correlation for Student Independent Use of the IWB
Pairs
Student Independent Use &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
1.00
(N=2)
--
Combined
.09
(N=75)
.452
(2-tailed)
Again, there were only two treatment/control studies in the continuation study which exhibited student
independent use of the IWB, one in each group. Because the two observations were at the opposite ends
of the scale, the correlation was 1.00. The combined finding reveals a small positive correlation between
student independent use and effect size which was not significant at the .05 level (p < .05). This would
suggest no linear relationship between effect size and student independent use of the IWB.
Evaluation Question 6: What is the relationship between teacher use of IWB reinforcers and
positive versus negative effects for the use of Promethean ActivClassroom?
IWB reinforcers refer to the use of those Promethean ActivClassroom tools that help indicate when a
correct answer has been produced or identified by students. High scores on this variable manifest as
frequent use of specific tools (e.g. drop and drag, applause when correct answer is identified or
produced). Figure 50 reports the difference in means between the positive-effects and negative-effects
classes for this variable.
Figure 50. Comparison of Means for Use of IWB Reinforcers
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
2
2
0
4
0
4
5
3.24
(N=17)
.53
Negative
0
1
0
0
0
3
4
2.00
(N=8)
.60
Positive
15
4
2
15
2
6
30
3.34
(N=74)
.28
13
2.17
(N=24)
.34
Continuation
Combined
Negative
0
2
1
3
0
5
t
Sig.
(2-tailed)
df
1.40
.174
23
2.65*
.011
55.33
* p < .05.
Final Report
48
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
The t-test for teacher use of IWB reinforcers in the continuation study shows that the positive-effects
group had a higher mean score than the negative-effects group. The difference in means between the two
groups was not significant (p < .05). The combined findings indicate a significant difference (p < .05)
between group means favoring the positive-effects group. The t-test for unequal variances was employed
because Levene’s test was significant (F = 10.71, p = .001).
The mean scores for both groups in the continuation study were below the middle value on the scale. In
the positive-effects group, the median was 2. In the negative-effects group, the median was 1.5. In both
groups, the mode was 1. Therefore, the measures of central tendency in both groups were at the low end
of the scale for this variable. When combined with the findings from the first year evaluation study, the
mean scores for both groups were also below the middle value on the scale. In the positive-effects
group, the median was 3 and the mode was 1. In the negative-effects group, the median and mode were
both 1. For both groups, the measures of central tendency were at the low end of the scale. This suggests
that most of the observations of teachers’ utilization of IWB reinforcers were lower than the middle
value of the scale.
Figure 51 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 51. Bivariate Correlation for Teacher Use of IWB Reinforcers
Pairs
IWB Reinforcers &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.45*
(N=25)
.025
Combined
.40***
(N=98)
.000
(2-tailed)
* p < .05; *** p < .001.
Figure 51 depicts a significant correlation (p < .05) between teacher use of IWB reinforcers and effect
size in the continuation study. This finding implies that as teachers’ utilization of tools that help to
indicate students’ correct answers increases, effect size tends to increase as well. The combined finding
also reveals a significant correlation (p < .001) suggesting that a strong positive linear relationship was
found between effect size and teachers’ utilization of IWB reinforcers.
Evaluation Question 7: What is the relationship between teacher use of learner response systems
and positive versus negative effects for the use of Promethean ActivClassroom?
Teacher use of learner response systems indicates the extent to which teachers have students answer
questions using one of Promethean’s learner response systems (ActiVote or ActivExpression). High
scores on this variable are indicated when teachers present students with questions and have each
student vote on the questions. Figure 52 reports the difference in means between the positive- and
negative-effects classes.
Final Report
49
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 52. Comparison of Means for Use of Learner Response Systems
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
1
3
3
5
0
6
2
3.70
(N=20)
.41
Negative
0
0
0
0
0
5
3
1.63
(N=8)
.18
Positive
18
4
3
13
0
7
32
3.42
(N=77)
.28
12
2.29
(N=24)
.37
Continuation
Combined
Negative
2
0
0
4
1
5
t
Sig.
(2-tailed)
df
4.61***
.000
24.65
2.40*
.020
51.96
* p < .05; *** p < .001.
In the continuation study, a significant difference (p < .001) was found between group means favoring
the positive-effects group. Levene’s test for equality of variances was significant (F = 11.31, p = .002).
When combined with the findings from the first year, a significant difference (p < .05) was also found
which favored the positive-effects group. As was the case for the continuation study, Levene’s test was
significant (F = 12.56, p = .001). Therefore, the t-test for unequal variances was performed for both the
continuation and combined studies.
The mean scores for both groups in the continuation study were below the middle value on the scale. In
the positive-effects group, the median was 4 and the mode was 2. Two of the three measures of central
tendency were near the middle of the scale for this variable. In the negative-effects group, the median
and mode were 2. Therefore, the measures of central tendency were at the low end of the scale.
The combined findings show that the mean scores were also below the middle value on the scale. In the
positive-effects group, the median was 2 and the mode was 1. In the negative-effects group, the median
was 1.5 and the mode was 1. The measures of central tendency for both groups were at the low end of
the scale. This seems to suggest that a majority of the observations of teachers’ utilization of learner
response systems were lower than the middle value of the scale.
Figure 53 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 53. Bivariate Correlation for Use of Learner Response Systems
Pairs
Learner Response Systems &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.49**
(N=28)
.008
Combined
.35***
(N=101)
.000
(2-tailed)
** p < .01; *** p < .001.
Final Report
50
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 53 shows that a significant correlation (p < .01) exists between teachers’ use of learner response
systems and effect size in the continuation study suggesting that as teachers’ utilization of learner
response systems increases, effect size tends to increase as well. When combined with the findings from
the first year, the correlation was lowered from .49 to .35. The correlation was significant at the .001
level (p < .001). Although reduced slightly, the finding suggests that a moderate positive linear
relationship was found between effect size and teachers’ utilization of learner response systems.
Evaluation Question 8: What is the relationship between teacher use of the IWB to represent
knowledge graphically or nonlinguistically and positive versus negative effects for the use of
Promethean ActivClassroom?
Representing knowledge graphically or nonlinguistically refers to use of the IWB to present content in
ways that do not employ language. High scores on this variable are indicated by use of pictures,
pictographs, graphic organizers and the like to present new information. Figure 54 reports the difference
in means between the positive-effects and negative-effects classes.
Figure 54. Comparison of Means for Use of the IWB for Nonlinguistic Representation
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
3
4
6
1
2
1
1
4.89
(N=18)
.40
Negative
1
0
2
0
2
3
0
3.63
(N=8)
.65
Positive
19
6
10
14
6
7
13
4.27
(N=75)
.25
6
3.21
(N=24)
.41
Continuation
Combined
Negative
2
2
3
2
4
5
t
Sig.
(2-tailed)
df
1.70
.102
24
2.11*
.037
97
* p < .05.
Figure 54 shows that in the continuation study, the positive-effects group exhibited a higher mean score
than the negative-effects group in terms of the utilization of the IWB for nonlinguistic representation.
The difference between group means was not significant at the .05 level (p < .05). The combined
findings reveal a significant difference (p < .05) between means favoring the positive-effects group.
In the continuation study, the mean score for the positive-effects group was above the middle value on
the scale. In the positive-effects group, both the median and mode were 5, indicating a central tendency
higher than the mean score of 4.89. In the negative-effects group, the median was 3 and the mode was 2.
Therefore, the measures of central tendency were at the low end of the scale.
In the combined continuation and first year studies, the mean score for the positive-effects group was
above the middle value on the scale. The median was 4 and the mode was 7. This indicates a high
central tendency in the positive-effects group. In the negative-effects group, the mean was below the
middle value on the scale. The median was 3 and the mode was 1. This suggests a low central tendency
Final Report
51
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
in the negative-effects group. A reasonable inference can be made that a difference was found between
the central tendencies of the observations of utilization of the IWB for nonlinguistic representation
which favors the positive-effects group.
Figure 55 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 55. Bivariate Correlation for Use of the IWB for Nonlinguistic Representation
Pairs
Nonlinguistic Representation &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.30
(N=26)
.143
Combined
.29**
(N=99)
.004
(2-tailed)
** p < .01.
The bivariate correlation for teachers’ use of the IWB for nonlinguistic representation was not
significant (p < .05) in the continuation study. When combined with the first year findings, the
correlation was significant at the .01 level (p < .01). The combined finding suggests that a moderate
positive linear relationship was found between effect size and teachers’ utilization of the IWB for
nonlinguistic representation.
Evaluation Question 9: What is the relationship between teacher previewing of content and
positive versus negative effects for the use of Promethean ActivClassroom?
Previewing of content refers to teachers providing students with advanced ways of thinking about a
topic before new information is presented. High scores on this variable are manifested by teachers
beginning a lesson with activities like: making linkages with what students already know about the
topic, providing some type of advance organizer for the content, etc. Figure 56 reports the difference in
means between the positive- and negative-effects classes.
Figure 56. Comparison of Means for Previewing of Content
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
0
3
3
7
3
2
0
4.11
(N=18)
.29
Negative
0
0
1
2
0
1
3
2.57
(N=7)
.65
Positive
9
8
5
19
5
15
14
3.61
(N=75)
.23
13
2.57
(N=23)
.45
Continuation
Combined
Negative
2
1
2
3
0
2
t
Sig.
(2-tailed)
df
2.51*
.019
23
2.16*
.034
96
* p < .05.
Final Report
52
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
The t-test for the continuation study indicates that the positive-effects group exhibited a higher mean
score than the negative-effects group. The difference between group means was statistically significant
at the .05 level (p < .05). The combined findings reveal a significant difference (p < .05) between group
means favoring the positive-effects group.
The mean score for the positive-effects group in the continuation study was close to the middle value on
the scale. In the positive-effects group, both the median and mode were 4, indicating a central tendency
at the middle of the scale. In the negative-effects group, the median was 2 and the mode was 4. In the
negative-effects group, the mean of 2.57 would be the best measure of central tendency.
The combined findings show that both group means were below the middle value on the scale. In the
positive-effects group, the median and mode were both 4. In the negative-effects group, both the median
and mode were 1. This would seem to suggest that there was a medium central tendency in the positiveeffects group and a low central tendency in the negative-effects group.
Figure 57 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 57. Bivariate Correlation for Previewing of Content
Pairs
Previewing of Content &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.63**
(N=25)
.001
Combined
.34**
(N=98)
.001
(2-tailed)
** p < .01.
Figure 57 reveals that a significant positive correlation (p < .01) exists between previewing of content
and effect size in the continuation study. This finding would seem to suggest that a strong positive linear
relationship was found between effect size and teachers providing students with opportunities to preview
content. When combined with the findings from the first year evaluation study, the correlation changed
from .63 to .34. The combined correlation was significant at the .01 level (p < .01) indicating that a
moderate positive linear relationship was found between effect size and teachers providing opportunities
for students to preview content.
Evaluation Question 10: What is the relationship between teacher chunking of content and
positive versus negative effects for the use of Promethean ActivClassroom?
Chunking refers to organizing new content in small segments. High scores on this variable are indicated
by teachers presenting content in “digestible bites” and then stopping so that students can process the
content. Figure 58 reports the difference in means between the positive- and negative-effects classes for
this variable.
Final Report
53
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 58. Comparison of Means for Teacher Chunking
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
0
3
5
7
1
3
0
4.21
(N=19)
.29
Negative
0
0
0
0
1
7
0
2.13
(N=8)
.13
Positive
9
9
19
18
4
14
3
4.30
(N=76)
.20
10
1.71
(N=24)
.14
Continuation
Combined
Negative
0
0
0
0
3
11
t
Sig.
(2-tailed)
df
6.57***
.000
23.20
10.80***
.000
91.92
*** p < .001.
Levene’s test for equality of variances was significant in both the continuation (F = 7.18, p = .013) and
combined studies (F = 15.88, p = .000). As such, Figure 58 reports t-tests for unequal variances. In the
continuation study, the t-test reveals a significant difference (p < .001) between mean scores on teacher
chunking favoring the positive-effects group. When combined with the first year findings, the difference
between means was also significant (p < .001) and favored the positive-effects group.
In the positive-effects group in the continuation study, both the median and mode were 4. All measures
of central tendency were at the middle of the scale for this variable. In the negative-effects group, the
median and mode were 2. All measures of central tendency were at the low end of the scale. In the
positive-effects group in the combined study, the median was 4 and the mode was 5. This would suggest
a central tendency just above the middle value on the scale. In the negative-effects group, both the
median and mode were 2 suggesting a central tendency at the low end of the scale. A reasonable
inference can be made that the positive-effects group exhibited more chunking than the negative-effects
group.
Figure 59 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 59. Bivariate Correlation for Teacher Chunking
Pairs
Chunking &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.70***
(N=27)
.000
Combined
.73***
(N=100)
.000
(2-tailed)
*** p < .001.
Figure 59 shows that a significant correlation (p < .001) exists between teacher chunking and effect size
in both the continuation and combined studies. These findings suggest that a strong positive linear
relationship was found between effect size and teachers’ organization of new content into small
segments.
Final Report
54
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 11: What is the relationship between teacher scaffolding of content and
positive versus negative effects for the use of Promethean ActivClassroom?
Scaffolding refers to the extent that “chunks” of information are presented in a way that one leads to
another. High scores on this variable are manifest as content being organized in a clear, logical
progression. Figure 60 reports the difference in means between the positive-effects and negative-effects
classes for this variable.
Figure 60. Comparison of Means for Teacher Scaffolding
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Positive
Mean
SE
7
6
5
4
3
2
1
2
1
6
5
2
3
0
4.32
(N=19)
.34
.38
Continuation
t
Negative
0
0
0
2
1
4
1
2.50
(N=8)
Positive
14
7
20
11
8
13
3
4.43
(N=76)
.21
8
2.25
(N=24)
.27
Combined
Negative
0
1
0
4
2
9
Sig.
(2-tailed)
df
3.11**
.005
25
6.40***
.000
52.53
** p < .01; *** p < .001.
The t-test for the continuation study indicates a significant (p < .01) difference between mean scores on
teacher scaffolding favoring the positive-effects group. When combined with the first year findings,
Levene’s test for equal variances was significant (F = 5.56, p = .020). As a result, the t-test for unequal
variances was reported. The combined findings show a significant (p < .001) difference between group
means which favors the positive-effects group.
In the continuation study, the median in the positive-effects group was 4 and the mode was 5. Therefore,
the measures of central tendency are just above the middle value on the scale. In the negative-effects
group, the median and mode were 2, indicating that the measures of central tendency were at the low
end of the scale. When the findings were combined with the first year evaluation study, both the median
and mode were 5 in the positive-effects group suggesting a high central tendency. In the negative-effects
group, the median and mode were 2 indicating a low central tendency. A reasonable inference can be
made that teachers in the positive-effects group utilized scaffolding more often than teachers in the
negative-effects group.
Figure 61 reports the bivariate correlation between this variable and the corrected effect sizes.
Final Report
55
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 61. Bivariate Correlation for Teacher Scaffolding
Pairs
Sig.
Evaluation Study
Correlation
Continuation
.73***
(N=27)
.000
Combined
.69***
(N=100)
.000
Scaffolding &
Corrected Effect Size
(2-tailed)
*** p < .001.
Figure 61 reveals significant (p < .001) positive correlations between scaffolding and effect size in both
the continuation and combined studies. These findings imply that a strong positive linear relationship
was found between effect size and teachers’ ability to present content that is organized in a clear, logical
progression.
Evaluation Question 12: What is the relationship between teacher pacing of content and positive
versus negative effects for the use of Promethean ActivClassroom?
Pacing of content refers to the speed with which the teacher guides students through new information.
High scores on this variable are manifest as the teacher slowing down or speeding up a presentation in
reaction to students’ needs and engagement levels. Figure 62 reports the difference in means between
positive- and negative-effects classes.
Figure 62. Comparison of Means for Pacing of Content
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
1
5
4
4
2
4
0
4.35
(N=20)
.36
Negative
0
0
0
1
3
2
2
2.38
(N=8)
.37
Positive
11
14
17
13
3
17
2
4.45
(N=77)
.21
10
2.08
(N=24)
.25
Continuation
Combined
Negative
0
0
1
3
3
7
t
Sig.
(2-tailed)
df
3.21**
.004
26
7.38***
.000
57.10
** p < .01; *** p < .001.
Figure 62 shows a significant difference (p < .01) in the continuation study between mean scores on
pacing of content that favors the positive-effects group. When combined with the findings from the first
year study, Levene’s test was significant (F = 7.89, p = .006). The t-test for unequal variances reveals
that the positive-effects group exhibited a higher mean score than the negative-effects group. The
difference between means was significant at the .001 level (p < .001).
Final Report
56
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
In the positive-effects group in the continuation study, the median was 4.5 and the mode was 6. This
would suggest that the central tendency was higher than the mean score of 4.35 for this group. In the
negative-effects group, the median was 2.5 and the mode was 3. This indicates that the central tendency
was higher than the mean score of 2.38 for this group.
In the combined findings, the positive-effects group had a median of 5 and two modes, 2 and 5. The
negative-effects group had a median and mode of 2. A reasonable inference can be made that teachers
adjusted their pacing more often in the positive-effects group than in the negative-effects group.
Figure 63 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 63. Bivariate Correlation for Pacing of Content
Pairs
Pacing of Content &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.63***
(N=28)
.000
Combined
.67***
(N=101)
.000
(2-tailed)
*** p < .001.
Figure 63 indicates that positive correlations exist between pacing of content and the corrected effect
sizes in both the continuation and combined studies. The bivariate correlations were statistically
significant at the .001 level (p < .001). These findings would seem to suggest that a strong positive linear
relationship was found between effect size and teachers’ adjustment of the pacing of their content in
response to students’ needs and engagement levels.
Evaluation Question 13: What is the relationship between teacher monitoring of student progress
and positive versus negative effects for the use of Promethean ActivClassroom?
Monitoring student progress refers to the teacher systematically identifying students’ level of
understanding of the content. High scores on this variable are manifest as the teacher informally or
formally assessing students and using the results to alter instruction. Figure 64 reports the difference in
means between the positive-effects class and the negative-effects class.
Final Report
57
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 64. Comparison of Means for Monitoring Student Progress
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
2
4
5
4
0
5
0
4.45
(N=20)
.38
Negative
0
0
0
0
1
4
3
1.75
(N=8)
.25
Positive
8
6
20
16
5
16
6
4.01
(N=77)
.20
14
1.67
(N=24)
.20
Continuation
Combined
Negative
0
0
0
2
2
6
t
Sig.
(2-tailed)
df
5.93***
.000
25.86
8.32***
.000
72.72
*** p < .001.
Levene’s test for equality of means in the continuation study was significant (F = 6.42, p = .018). The ttest for unequal variances indicates that the positive-effects group exhibited a higher mean score on this
variable. The difference between means was significant at the .001 level (p < .001). Levene’s test in the
combined study was also significant (F = 9.38, p = .003). The t-test for unequal variances shows a
significant difference (p < .001) between mean scores which favors the positive-effects group.
In the continuation study, there were two modes (2 and 5) in the positive-effects group with a median of
5. The measures of central tendency were higher than the middle value on the scale. In the negativeeffects group, the median and mode were 2. The measures of central tendency were lower than the
middle value on the scale. When the findings were combined with the first year study, the mode was 5 in
the positive-effects group with a median of 4. In the negative-effects group, both the median and mode
were 1. A reasonable inference can be made that monitoring of student progress took place more
frequently in the positive-effects group than it did in the negative-effects group.
Figure 65 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 65. Bivariate Correlation for Monitoring Student Progress
Pairs
Monitoring Student Progress &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.79***
(N=28)
.000
Combined
.72***
(N=101)
.000
(2-tailed)
*** p < .001.
Figure 65 reveals that positive correlations exist between monitoring student progress and the corrected
effect sizes in both the continuation and combined studies. The correlations were statistically significant
at the .001 level (p < .001). These findings would seem to suggest that a strong positive linear
relationship was found between effect size and teachers’ utilization of the results of formal and informal
assessments to alter instruction.
Final Report
58
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 14: What is the relationship between clarity of content as depicted in the
IWB and positive versus negative effects for the use of Promethean ActivClassroom?
Clarity of content as depicted on the IWB involves the teacher designing flipcharts so that important
content is made clear. High scores on this variable are manifest as the teacher keeping each flipchart free
of distracting content and images or the teacher using highlighting features to mark the critical versus
peripheral content. Figure 66 reports the difference in means between the positive-effects class and the
negative-effects class for this variable.
Figure 66. Comparison of Means for Clarity of Content on the IWB
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
3
1
5
4
1
4
0
4.39
(N=18)
.41
Negative
0
0
0
1
1
5
1
2.25
(N=8)
.31
Positive
13
9
21
12
7
9
4
4.55
(N=75)
.20
8
2.21
(N=24)
.27
Continuation
Combined
Negative
0
1
0
4
1
10
t
Sig.
(2-tailed)
df
4.18***
.000
23.24
6.93***
.000
51.69
*** p < .001.
In the continuation study, Levene’s test for equality of means was significant (F = 4.33, p = .048). The ttest for unequal variances shows that the positive-effects group had a higher mean score. The difference
between group means was significant at the .001 level (p < .001). When combined with the first year
findings, Levene’s test was significant (F = 4.44, p = .038). The difference between group means
favored the positive-effects group and was statistically significant (p < .001).
In the positive-effects group in the continuation study, the median was 4.5 and the mode was 5. The
median and mode were higher than the mean of 4.39 suggesting a central tendency higher than the
middle value of the scale. In the negative-effects group, the median and mode were 2. All of the
measures of central tendency were lower than the middle value of the scale.
In the combined findings, the median of the positive-effects group was 5 and the mode was also 5. In the
negative-effects group, the median and mode were 2. This seems to suggest that the measures of central
tendency were high in the positive-effects group and low in the negative-effects group. A reasonable
inference can be made that clarity of content was observed more frequently in the positive-effects group.
Figure 67 reports the bivariate correlation between this variable and the corrected effect sizes.
Final Report
59
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 67. Bivariate Correlation for Clarity of Content on the IWB
Pairs
Clarity of Content &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.76***
(N=26)
.000
Combined
.67***
(N=99)
.000
(2-tailed)
*** p < .001.
In both the continuation and combined studies, the bivariate correlations between clarity of content and
effect size were statistically significant at the .001 level (p < .001). These findings would appear to
suggest that a strong positive linear relationship was found between effect size and the clarity of content
presented with Promethean ActivClassroom.
Evaluation Question 15: What is the relationship between student interaction about the content
and positive versus negative effects for the use of Promethean ActivClassroom?
Student interaction about the content involves students discussing the content with one another. High
scores on this variable are indicated when the teacher includes activities like: students answering
questions in groups, students summarizing new content in groups, students making predictions in
groups, etc. Figure 68 reports the difference in means between the positive- and negative-effects classes.
Figure 68. Comparison of Means for Student Interaction
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
0
2
3
4
2
6
3
3.20
(N=20)
.37
Negative
0
0
0
0
0
6
2
1.75
(N=8)
.16
Positive
7
5
5
15
4
16
25
3.03
(N=77)
.23
12
1.96
(N=24)
.27
Continuation
Combined
Negative
0
1
0
3
1
7
t
Sig.
(2-tailed)
df
3.61**
.001
24.66
3.00**
.004
58.32
** p < .01.
Figure 68 reveals a significant (p < .01) difference between group means favoring the positive-effects
group in the continuation study. Levene’s test for equality of means was significant (F = 14.43, p =
.001). Therefore, the t-test for unequal variances was reported. When combined with the first year
findings, Levene’s test was significant as well (F = 12.11, p = .001). The t-test for unequal variances
indicates that the difference between means favored the positive-effects group. The difference was
significant at the .01 level (p < .01).
Final Report
60
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
In the continuation study, the positive-effects group had a median of 3 and a mode of 2. The median and
mode were lower than the mean of 3.20. In the negative-effects group, the median and mode were 2. The
median and mode were higher than the mean of 1.75. When combined with the first year findings, the
positive-effects group had a median of 2 and a mode of 1. The negative-effects group had a median of
1.5 and a mode of 1. Although the difference between means favored the positive-effects group, all of
the measures of central tendency in both groups were lower than the middle value of the scale.
Figure 69 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 69. Bivariate Correlation for Student Interaction
Pairs
Student Interaction &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.57**
(N=28)
.002
Combined
.39***
(N=101)
.000
(2-tailed)
** p < .01; *** p < .001.
Figure 69 shows a significant correlation (p < .01) between student interaction and effect size in the
continuation study. When combined with the findings from the first year study, the correlation changed
from .57 to .39. The correlation was statistically significant (p < .001). This finding seems to suggest
that a moderate positive linear relationship was found between effect size and student discussion of
content.
Evaluation Question 16: What is the relationship between student response rates and positive
versus negative effects for the use of Promethean ActivClassroom?
Student response rates refer to the extent to which students respond to direct queries from the teacher.
High scores on this variable manifest as multiple students responding to teacher questions (as opposed to
a small group of students) and responses being elicited by the teacher frequently throughout a class
period. Figure 70 reports the difference in means between the positive-effects and negative-effects
classes for this variable.
Final Report
61
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 70. Comparison of Means for Student Response Rate
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
3
1
5
4
3
4
0
4.25
(N=20)
.38
Negative
0
0
0
0
1
4
3
1.75
(N=8)
.25
Positive
11
7
10
18
11
16
4
4.03
(N=77)
.21
14
1.88
(N=24)
.28
Continuation
Combined
Negative
0
1
1
1
2
5
t
Sig.
(2-tailed)
df
5.54***
.000
25.82
5.35***
.000
99
*** p < .001.
In the continuation study, Levene’s test was significant (F = 5.80, p = .023). The t-test for unequal
variances shows that a significant difference (p < .001) exists between group means favoring the
positive-effects group. The combined findings reveal a significant difference (p < .001) between group
mean scores which favors the positive-effects group.
In the continuation study, the median in the positive-effects group was 4 and the mode was 5. Two of
the three measures of central tendency were above the middle value on the scale. In the negative-effects
group, the median and mode were 2. All three measures of central tendency were lower than the middle
value of the scale.
In the combined findings, the median and mode in the positive-effects group were both 4. In the
negative-effects group, the median and mode were 1. These findings suggest a central tendency near the
middle of the scale in the positive-effects group and a central tendency at the low end of the scale in the
negative-effects group.
Figure 71 reports the bivariate correlation between this variable and the corrected effect sizes.
Figure 71. Bivariate Correlation for Student Response Rate
Pairs
Student Response Rate &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.73***
(N=28)
.000
Combined
.67***
(N=101)
.000
(2-tailed)
*** p < .001.
Figure 71 reveals a significant correlation (p < .001) between student response rate and effect size in
both the continuation and combined studies. These findings imply that a strong positive linear
relationship was found between effect size and student response rate.
Final Report
62
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 17: What is the relationship between classroom management and positive
versus negative effects for the use of Promethean ActivClassroom?
Classroom management refers to the extent to which the teacher uses management strategies that foster
a safe and orderly classroom environment. High scores on this variable are manifest as teachers having
clear rules and procedures and responding appropriately to classroom interruptions and violations of
rules and procedures. Figure 72 reports the difference in means between the positive- and negativeeffects classes.
Figure 72. Comparison of Means for Classroom Management
Evaluation
Study
Frequencies
T-Test for Equality of Means
Effects
Mean
SE
7
6
5
4
3
2
1
Positive
5
5
7
2
0
0
0
5.68
(N=19)
.23
Negative
1
2
4
1
0
0
0
5.38
(N=8)
.32
Positive
28
13
21
13
0
0
1
5.68
(N=76)
.14
0
5.08
(N=24)
.31
Continuation
Combined
Negative
6
4
5
5
3
1
t
Sig.
(2-tailed)
df
.75
.461
25
1.94
.056
98
As shown in Figure 72, the t-test for the continuation study reveals that the positive-effects group
exhibited a higher mean score for classroom management than the negative-effects group. The
difference between group means was not statistically significant at the .05 level (p < .05). The combined
findings also favor the positive-effects group with a difference between means that approaches
significance at the .05 level (p < .05).
In the positive-effects group in the continuation study, the median was 6 and the mode was 5. The mean
score was between the median and mode. In the negative-effects group, the median and mode were 5.
The median and mode were below the mean score of 5.38. All measures of central tendency in both
groups were higher than the middle value on the scale. When combined with the first year findings, the
median in the positive-effects group was 6 and the mode was 7. In the negative-effects group, the
median was 5 and the mode was 7. This would seem to suggest that both groups exhibited a high central
tendency with respect to teachers having clear rules and procedures and responding appropriately to
classroom interruptions and violations of rules and procedures.
Figure 73 reports the bivariate correlation between this variable and the corrected effect sizes.
Final Report
63
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 73. Bivariate Correlation for Classroom Management
Pairs
Classroom Management &
Corrected Effect Size
Sig.
Evaluation Study
Correlation
Continuation
.27
(N=27)
.169
Combined
.30**
(N=100)
.002
(2-tailed)
** p < .01.
The bivariate correlations in the continuation study between classroom management and the corrected
effect sizes were not significant at the .05 level (p < .05). The combined findings reveal a significant
positive correlation (p < .01). A reasonable inference can be made that a moderate positive linear
relationship was found between effect size and classroom management.
Interpretation
The results from the Phase II analysis provide more detail as to the relationship between the utilization
of Promethean ActivClassroom and student achievement. This section focuses on the combined findings
from the first year and continuation studies. Four general areas of interest are considered.
Comparison of Means versus Bivariate Correlation
First, it is worth noting that 15 of the 17 Phase II variables exhibited correlations with the corrected
effect sizes from Phase I that were statistically significant at the .05 level (p < .05). It should be further
noted that the correlations for all 15 variables would also be considered significant at the .01 level (p <
.01). In addition, 12 of the 15 correlations would be considered significant at the .001 level (p < .001).
Student IWB skills (Variable 3) and student independent use of the IWB (Variable 5) did not have
significant correlations.
Of the 15 Phase II variables with significant correlations to effect size, only classroom management
(Variable 17) did not exhibit a statistically significant difference on the t-test for equality of means
between positive-effects and negative-effects classes (t = 1.94, df = 98, p > .05). However, it should be
noted that the t-test approached significance (p = .056) at the .05 level (p < .05).
Engagement and Student Achievement
Second, the correlation between engagement (Variable 1) and corrected effect size (r = .44, N = 101, p <
.001) is consistent with the general finding in Phase I that the use of Promethean ActivClassroom was
related to student achievement (see Figure 19). Figure 40 shows that the positive-effects classes had
significantly higher mean scores on engagement than the negative-effects classes (t = 3.43, df = 99, p <
.01). The measures of central tendency reveal that the positive-effects group had relatively high
engagement scores and the negative-effects group had relatively moderate engagement scores (see
discussion of Figure 40). Although a significant difference between mean scores on engagement was
Final Report
64
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
found, two of the three measures of central tendency in the negative-effects group were above the
middle value on the scale suggesting that, in general, students were engaged in both types of classes.
Variables 6, 7, and 8
Third, variables 6, 7, and 8 offer insight into the effects of specific features of Promethean
ActivClassroom.
Variable 6 addressed the extent to which teachers used Promethean ActivClassroom tools to reinforce
correct answers produced by students. One interpretation of the correlation between this variable and
corrected effect size (r = .40, N = 98, p < .001) is that if a typical teacher’s use of this tool were to
increase by one standard deviation, the corrected effect size would be predicted to increase by 16
percentile points on the distribution of corrected effect sizes.
Variable 7 addressed the extent to which teachers used learner response systems. One interpretation of
the correlation between this variable and corrected effect size (r = .35, N = 101, p < .001) is that if a
typical teacher’s use of this tool were to increase by one standard deviation, the corrected effect size
would be predicted to increase by 14 percentile points on the distribution of corrected effect sizes.
Variable 8 addressed the extent to which teachers used Promethean ActivClassroom to represent
knowledge graphically or nonlinguistically. One interpretation of the correlation between this variable
and corrected effect size (r = .29, N = 99, p < .01) is that if a typical teacher’s use of this tool were to
increase by one standard deviation, the corrected effect size would be predicted to increase by 11
percentile points on the distribution of corrected effect sizes.
Variables 10, 11, 12, 13, 14, and 16
Fourth, an examination of the effects of variables 10, 11, 12, 13, 14, and 16 provides insight into general
instructional variables that might interact with use of Promethean ActivClassroom. All of these variables
exhibited correlations with effect size that were greater than .60 and could be considered large effects in
the social sciences (see Lipsey & Wilson, 2001, p. 147).
Variable 10 addressed chunking of content. One interpretation of the correlation between this variable
and corrected effect size (r = .73, N = 100, p < .001) is that if a typical teacher’s use of this tool were to
increase by one standard deviation, the corrected effect size would be predicted to increase by 27
percentile points on the distribution of corrected effect sizes.
Variable 11 addressed scaffolding. One interpretation of the correlation between this variable and
corrected effect size (r = .69, N = 100, p < .001) is that if a typical teacher’s use of this tool were to
increase by one standard deviation, the corrected effect size would be predicted to increase by 26
percentile points on the distribution of corrected effect sizes.
Final Report
65
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Variable 12 addressed pacing. One interpretation of the correlation between this variable and corrected
effect size (r = .67, N = 101, p < .001) is that if a typical teacher’s use of this tool were to increase by
one standard deviation, the corrected effect size would be predicted to increase by 25 percentile points
on the distribution of corrected effect sizes.
Variable 13 addressed monitoring of student progress. One interpretation of the correlation between this
variable and corrected effect size (r = .72, N = 101, p < .001) is that if a typical teacher’s use of this tool
were to increase by one standard deviation, the corrected effect size would be predicted to increase by
26 percentile points on the distribution of corrected effect sizes.
Variable 14 addressed the clarity of the content as depicted on the IWB. One interpretation of the
correlation between this variable and corrected effect size (r = .67, N = 99, p < .001) is that if a typical
teacher’s use of this tool were to increase by one standard deviation, the corrected effect size would be
predicted to increase by 25 percentile points on the distribution of corrected effect sizes.
Variable 16 addressed student response rates. One interpretation of the correlation between this variable
and corrected effect size (r = .67, N = 101, p < .001) is that if a typical teacher’s use of this tool were to
increase by one standard deviation, the corrected effect size would be predicted to increase by 25
percentile points on the distribution of corrected effect sizes.
Figure 74 displays the product moment correlations between these variables.
Final Report
66
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Figure 74. Product Moment Correlations between Variables 10, 11, 12, 13, 14, and 16
Comparison
Correlation
Variable 10
Variable 11
Variable 12
Variable 13
Variable 14
Variable 16
Variable
10
Variable
11
Variable
12
Variable
13
Variable
14
Variable
16
1
.86***
.81***
.78***
.81***
.72***
.000
.000
.000
.000
.000
Sig. (2-tailed)
N
100
100
100
100
99
100
Correlation
.86***
1
.78***
.74***
.82***
.65***
Sig. (2-tailed)
.000
.000
.000
.000
.000
N
100
100
100
100
99
100
Correlation
.81***
.78***
1
.71***
.76***
.66***
Sig. (2-tailed)
.000
.000
.000
.000
.000
N
100
100
101
101
99
101
Correlation
.78***
.74***
.71***
1
.66***
.70***
Sig. (2-tailed)
.000
.000
.000
.000
.000
N
100
100
101
101
99
101
Correlation
.81***
.82***
.76***
.66***
1
.61***
Sig. (2-tailed)
.000
.000
.000
.000
N
99
99
99
99
99
99
Correlation
.72***
.65***
.66***
.70***
.61***
1
Sig. (2-tailed)
.000
.000
.000
.000
.000
N
100
100
101
101
99
.000
101
*** p < .001.
Figure 74 indicates that these variables are highly intercorrelated. Stated differently, the high
correlations imply that these instructional practices are somehow linked in terms of their implementation
in the classroom.
In order to assess their composite effect, a macro written for meta-regression in SPSS was used to
conduct a modified weighted least squares regression analysis (Wilson, 2005). This approach was taken
because the weighting procedure employed by SPSS makes an assumption that the weights represent
different subjects as opposed to studies, and as such, statistical significance tests are based on sample
size assumptions that would not apply to meta-analytic data (see Lipsey & Wilson, 2001). A mixedeffects model was utilized in the meta-regression analysis with a maximum likelihood estimate of the
random-effects variance component.
Final Report
67
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
The corrected effect sizes from Phase I were the dependent variable in the meta-regression and the
ratings on chunking (Variable 10), scaffolding (Variable 11), pacing (Variable 12), monitoring (Variable
13), clarity of IWB (Variable 14), and student response rate (Variable 16) were the independent
variables. The inverse of the variance (1/SE2) was used to weight the corrected effect sizes. The results
of this analysis are depicted in Figure 75.
Figure 75. Weighted Meta-Regression Analysis with Corrected Effect Size as the Dependent
Variable and Variables 10, 11, 12, 13, 14, and 16 as Independent Variables
R
R2
Adj. R2
N
.789
.622
.597
99
Q
df
Sig.
(2-tailed)
Model
158.745****
6
.0000
Residual
96.563
92
.3520
Total
255.308****
98
.0000
Model Summary
v = .06; se(v) = .02. **** p < .0001.
Figure 75 shows that the overall model was statistically significant at the .0001 level (p < .0001).
Applying a widely used convention for appraising the magnitude of effect sizes, the multiple correlation
between the observed values of the dependent variable (i.e., corrected effect size) and the predicted
values (R = .789, N = 99, p < .0001) would be considered very large in the social sciences (see Lipsey &
Wilson, 2001, p. 147). Approximately 62% of the variability in the dependent variable was accounted
for by the variables in this model (R2 = .622). After taking into account the number of predictors, about
60% of the variability was accounted for by the variables in the model (Adj. R2 = .597) with about 40%
of the variance attributable to other factors not examined in this report.
Phase II Summary and Interpretation
Phase II of the evaluation study regarding the effects of Promethean ActivClassroom on student
achievement sought to answer the following questions:
Evaluation Question 1: What is the relationship between student engagement and positive versus
negative effects for the use of Promethean ActivClassroom?
Evaluation Question 2: What is the relationship between teacher IWB skills and positive versus negative
effects for the use of Promethean ActivClassroom?
Evaluation Question 3: What is the relationship between student IWB skills and positive versus negative
effects for the use of Promethean ActivClassroom?
Evaluation Question 4: What is the relationship between multiple student use of the IWB and positive
versus negative effects for the use of Promethean ActivClassroom?
Final Report
68
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Evaluation Question 5: What is the relationship between student independent use of the IWB and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 6: What is the relationship between teacher use of IWB reinforcers and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 7: What is the relationship between teacher use of learner response systems and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 8: What is the relationship between teacher use of the IWB to represent knowledge
graphically or nonlinguistically and positive versus negative effects for the use
of Promethean ActivClassroom?
Evaluation Question 9: What is the relationship between teacher previewing of content and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 10: What is the relationship between teacher chunking of content and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 11: What is the relationship between teacher scaffolding of content and positive
versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 12: What is the relationship between teacher pacing of content and positive versus
negative effects for the use of Promethean ActivClassroom?
Evaluation Question 13: What is the relationship between teacher monitoring of student progress and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 14: What is the relationship between clarity of content as depicted in the IWB and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 15: What is the relationship between student interaction about the content and
positive versus negative effects for the use of Promethean ActivClassroom?
Evaluation Question 16: What is the relationship between student response rates and positive versus
negative effects for the use of Promethean ActivClassroom?
Evaluation Question 17: What is the relationship between classroom management and positive versus
negative effects for the use of Promethean ActivClassroom?
Phase II involved an analysis of student learning with and without Promethean ActivClassroom from the
perspective of teacher behavior as evidenced in videotapes of teachers using the technology in their
classrooms. The general focus of Phase II was to determine the behaviors that differentiated those
teachers who obtained positive effects from Promethean ActivClassroom from those who did not.
The combined findings from the first year and continuation studies can be summarized in the following
statements:
•
•
The correlations between 15 of the 17 variables addressed in Phase II and the corrected effect sizes
calculated in Phase I were statistically significant at the .05 level (p < .05). The correlations for all
15 variables would also be considered significant at the .01 level (p < .01). In addition, 12 of the 15
correlations would be considered significant at the .001 level (p < .001). Student IWB skills
(Variable 3) and student independent use of the IWB (Variable 5) did not have significant
correlations.
Of the 15 Phase II variables with significant correlations to effect size, only classroom management
(Variable 17) did not exhibit statistical significance on the t-test for equality of means at the .05 level
(p < .05). However, the t-test did approach significance (t = 1.94, df = .98, p = .056).
Final Report
69
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
•
•
The correlations between 14 of the 17 Phase II variables and the corrected effect sizes were greater
than .30. Of those 14 variables, 6 exhibited correlations with corrected effect size that were greater
than .60, which would seem to suggest that substantial increases in student achievement would be
predicted with improvements in teacher behavior with respect to chunking (Variable 10), scaffolding
(Variable 11), pacing (Variable 12), monitoring (Variable 13), clarity of IWB (Variable 14), and
student response rate (Variable 16).
The multiple correlation of .789 (N = 99, p < .0001) reported in Figure 75 might suggest a strong
effect on student achievement when the following conditions are met:
o New content is organized into small digestible bites designed with students’ background
knowledge and needs in mind (chunking).
o Chunks of new content follow a logical progression so that each chunk helps students
understand the next (scaffolding).
o The pace at which each chunk is addressed is adjusted as needed (i.e., slower, faster) to
maintain high engagement and comprehension (pacing).
o Students’ ability to understand new content is consistently monitored (monitoring).
o Promethean ActivClassroom is utilized so that essential content is presented in a clear
manner. Content should be reviewed or re-taught when it becomes apparent that students
misunderstand portions of the content (clarity of IWB).
o Questions are asked and addressed in a manner that would allow all students to have an
opportunity to respond. Students’ answers should be continually examined for correctness
and depth of understanding (student response rate).
Final Report
70
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Technical Notes
Technical Note 1: Conceptually, analysis of covariance (ANCOVA) can be thought of as using a
covariate to predict performance on the dependent variable and then using the residual scores (predicted
score minus observed score) as the dependent measure in the analysis. For this report, students’ posttest
scores were predicted using the scores received on the pretest. The difference between the predicted
posttest scores and the observed posttest scores were then computed for each student that took both
pretest and posttest. This difference is referred to as the residual score for each student. It represents the
part of each student’s posttest score that cannot be predicted from the pretest score for that student.
Theoretically, use of residual scores based on pretest predictions is an attempt to equate all students on
the dependent measure prior to execution of the intervention—in this case the use of Promethean
ActivClassroom.
Technical Note 2: The quasi-experimental design used in the treatment/control studies is referred to as a
pretest-posttest non-equivalent groups design. The groups are considered to be non-equivalent because
students were not assigned to treatment and control conditions through random lottery. Pretest scores
were used as a covariate to statistically equate the students and partially control for differing levels of
background knowledge and skill. Although ANCOVA was used to statistically equate students in terms
of prior achievement, without randomization arguments about causal relationships are severely
weakened.
While quasi-experimental designs lack the rigor of fully randomized experiments, the pretest-posttest
nonrandomized research design has a natural fit with classroom teachers. This design allows teachers to
use one intact class as a control group and a second intact class as the treatment group. Potential
concerns over making adjustments to existing lesson plans or instructional time in order to isolate
control group students from treatment group students in a single class would not apply. In other words,
the use of this quasi-experimental design minimizes the potential impact on a teacher’s normal class
routine. An assumption was made that both groups in the treatment/control studies shared a similar
demographic (e.g., ethnicity, socio-economic status, etc.) which would allow for a fair comparison. This
notwithstanding, random assignment of students to treatment and control groups is always preferable in
evaluation studies. This effort does not meet that criterion.
Technical Note 3: The generic term effect size applies to a variety of indices (e.g., d, r, R, PV) that can
be used to demonstrate the effect of an independent variable (e.g., use of Promethean ActiVote) on a
dependent variable (e.g., student academic achievement). As used in this report, effect size refers to the
standardized mean difference effect size. This index, first popularized by Glass (1976) and Cohen
(1977), is the difference between experimental and control group means divided by an estimate of the
population standard deviation.
mean of treatment group – mean of control group
standardized mean difference effect size
=
estimate of population standard deviation
Final Report
71
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Consider the following illustration of the use of effect size. Assume that the achievement mean of a
group of students in a class that used a specific instructional strategy (e.g., graphic organizers) is 90 on a
standardized test and the mean of a group of students in a class that did not use the instructional strategy
is 80. Assuming the population standard deviation is 10, the effect size would be as follows:
𝐸𝑆 =
90 − 80
= 1.0
10
Thus, an effect size of 1.0 would indicate that the mean score in the treatment group is one standard
deviation higher than the mean score in the control group. Conversely, an effect size of -1.0 would
indicate that the mean score in the treatment group is one standard deviation lower than the mean score
in the control group.
Technical Note 4: The meta-analytic findings in this report are typically reported in two ways—
observed and corrected. The corrected findings have been corrected for attenuation due to a lack of
reliability in the dependent measure (i.e., teacher-designed assessments of student academic
achievement). Hunter and Schmidt detail the rationale and importance of correcting for 11 attenuation
artifacts—one of which is random error associated with measurement of the dependent variable (2004,
pp. 301-313). They explain:
. . . error of measurement in the dependent variable reduces the effect size estimate. If the
reliability of measurement is low, the reduction can be quite sizable. Failure to correct for the
attenuation due to error of measurement yields an erroneous effect size estimate. Furthermore,
because the error is systematic, a bare-bones meta-analysis on uncorrected effect sizes will
produce an incorrect estimate of the true effect size. The extent of the reduction in the mean
effect size is determined by the mean level of reliability across the studies. Variation in reliability
across studies causes variation in the observed effect size above and beyond that produced by
sampling error. . . . A bare-bones meta-analysis will not correct for either the systematic
reduction in the mean effect size or the systematic increase in the variance of effect sizes. Thus,
even meta-analysis will produce correct values for the distribution of effect sizes only if there is
a correction for the attenuation due to error of measurement. (p. 302)
For ease of discussion we consider correcting for attenuation due to unreliability in the dependent
measure using the population correlation instead of the population standardized mean difference effect
size. The reader should note that the example provided regarding correcting correlations is analogous to
correcting a standardized mean difference. To illustrate correcting for attenuation due to unreliability in
the dependent measure, assume that the population correlation between a specific instructional strategy
(e.g., nonlinguistic representations) and student academic achievement is .50. A given study attempts to
estimate that correlation but employs a measure of the dependent variable (i.e., a teacher-designed
assessment of student academic achievement) that has a reliability of .81—considered a typical
reliability for a test of general cognitive ability. According to attenuation theory, the correlation would
be reduced by the square root of the reliability (i.e., the attenuation factor). In other words, the
population correlation is multiplied by the attenuation factor (√. 81 = .90), thus reducing the correlation
by 10 percent. Therefore, the observed correlation will be .45 (.50 x .90) even if there is no attenuation
due to the other ten artifacts listed by Hunter and Schmidt (2004, p. 35). When the measure of the
Final Report
72
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
dependent variable has a lower reliability, .36 for example, the correlation is reduced by 40 percent
(√. 36 = .60) to .30 (.50 x .60). In order to make a correction for attenuation, the correlation is divided
by the attenuation factor (i.e., the square root of the reliability). (For a more detailed discussion of
attenuation in the context of meta-analysis see Hunter and Schmidt, 2004.)
For the purposes of this report, an estimate of reliability was used. Osborne, Christensen, and Gunter
reported that the average reliability reported in top Educational Psychology journals was .83 (as cited by
Osborne, 2003). Lou and colleagues (1996) reported a typical reliability of .85 for standardized
achievement tests and a reliability of .75 for unstandardized achievement tests. Because the dependent
measure in the independent action research studies involved teacher-designed assessments of student
academic achievement, .75 was used as the reliability to correct for attenuation using the following
formula:
𝑑𝑐 =
𝑑𝑜
𝑎
In the formula, 𝑑𝑐 is the corrected effect size, 𝑑𝑜 is the observed effect size, and 𝑎 is the attenuation
factor. Using this formula, each effect size reported in Figure 18 was corrected for attenuation to
produce the corrected meta-analytic findings considered in this report.
Technical Note 5: Independent variables can be analyzed as fixed effects or as random effects. In the
context of ANOVA/ANCOVA, fixed effects are factors with levels that are deliberately arranged by the
researcher. In contrast, random effects are factors with levels that are not deliberately arranged. Instead,
random effects are factors which are randomly sampled from a population of possible samples.
Generally speaking, when independent variables are analyzed as random effects, the intent is to
generalize results beyond the boundaries of the independent variables employed in the study. For
example, if a researcher were interested in the effect that the quality of school leadership has on
academic proficiency, the researcher could select a random sample of schools in order to estimate the
amount of variance in student academic achievement attributable to differences between types of school
leaders. Thus, using the sample, the researcher can make generalizations regarding the influence of
school leadership on academic achievement as a whole. Additional research could attempt to replicate
the findings by duplicating the study and selecting a different random sample of schools for comparison.
If the findings could be replicated then the generalization would have increased validity. However, if the
findings could not be replicated then the generalization might not be valid across all settings. An
alternative hypothesis might need to be explored. When fixed effects are employed one typically does
not generalize beyond the boundaries of the independent variables in the study. However, additional
research could still attempt to generalize the findings by replicating every facet of the study. For
example, multiple studies could be used to determine the influence of a specific instructional technique
in the classroom.
Final Report
73
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Technical Note 6: In Figure 18, the column labeled “ES” contains the computed effect size for each
study calculated as Cohen’s d using the following formula:
𝑑=
𝑟
�(1 − 𝑟 2 )(𝑝(1 − 𝑝))
In the formula r is the effect size correlation (e.g., point-biserial correlation coefficient) and p is the
proportion of the total population in one of the two groups (i.e., the treatment group). This formula is
used to compute the standardized mean difference effect size from an effect size correlation when the
treatment and control group populations are not equal (see Lipsey & Wilson, 2001, pp. 62-63).
ANCOVA was employed for each treatment/control study. Partial eta squared (η2𝑝 ) is a measure of
strength of association between independent variables generated by SPSS for the ANCOVA (SPSS,
2009). Another measure of strength of association, eta squared (η2 ), is commonly used to describe the
proportion of total variability in a dependent variable (e.g., student’s posttest scores) that can be
explained by an independent variable (e.g., the use of Promethean ActivClassroom). Whereas, eta
squared (η2 ) describes the proportion of the total variance attributable to an effect, partial Eta squared
(η2𝑝 ) describes the proportion of the effect plus error that is attributable to the effect. These two measures
are often depicted by the following formulas:
𝜂2 =
𝑆𝑆𝑒𝑓𝑓𝑒𝑐𝑡 2
𝑆𝑆𝑒𝑓𝑓𝑒𝑐𝑡
, 𝜂𝑝 =
𝑆𝑆𝑡𝑜𝑡𝑎𝑙
𝑆𝑆𝑒𝑓𝑓𝑒𝑐𝑡 + 𝑆𝑆𝑒𝑟𝑟𝑜𝑟
In the formulas, SSeffect is the sum of squares for whatever effect is of interest, SSerror is the sum of
squares for whatever error term is associated with that effect, and SStotal is the total sums of squares for
all effects of interest, interactions, and errors in the ANCOVA. When there is one degree of freedom, the
square root of eta squared (η2 ) is r. Because only one independent variable was considered in the
ANCOVA, the square root of partial eta squared (η2𝑝 ) was used as a reasonable estimate for r in the
formula for computing d above.
Technical Note 7: The standardized mean difference effect size (Cohen’s d) expresses the differences
between means in standardized or “Z score” form, which gives rise to another index frequently used in
research regarding education—percentile gain.
By way of illustration, consider the example in Technical Note 3. An effect size (Cohen’s d) of 1.0 can
be interpreted as the average score in the treatment group being about 34 percentile points greater than
the average score in the control group. It is important to note that this does not mean that the average
score in the treatment group is 34% greater than the average score in the control group. Instead, it refers
to the percentile point on the distribution of scores. For example, a score at the 65th percentile would be
the lowest score that is greater than 65% of the scores in the distribution. Again, the effect size translates
the difference between group means into Z score form. Distribution theory dictates that a Z score of 1.0
is at the 84.13 percentile point of the standard normal distribution. To determine the percentile gain, the
effect size is transformed into percentile points above or below the 50th percentile point on the unit
Final Report
74
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
normal distribution (e.g., 84.1 - 50.0 = 34.1). The percentile gain values listed in this report were
determined by consulting a normal curve table for the area for each reported effect size.
Using the current example of a study with an effect size (Cohen’s d) of 1.0, the average student in the
treatment group would be expected to be at the 84th percentile in the control group, which is 34
percentile points greater than the average student in the control group. Stated differently, the average
student in the treatment group (i.e., the student at the 50th percentile) scored better than 84% of the
students in the control group. An alternate interpretation involves a student who is ranked in the middle
of a control group with 100 students. If this student were the only one to receive the treatment, his or her
ranking would improve from 50th to 17th. In other words, the student would be expected to surpass 33%
of the students that did not receive the treatment.
Sometimes it can be helpful to visualize the effect size in terms of the overlapping distributions of scores
from the two groups. One of the most widely used approaches is the binomial effect size display
(BESD). This approach assumes a success threshold at the median of the distribution of scores on a
dependent variable for the entire population (i.e., combined treatment and control groups). This
threshold is always set at 50% for the combined distributions. The BESD can be used to show the
proportion of students who would be expected to pass or fail a hypothetical test represented by the
dependent variable assuming an initial passing rate of .50 for all students. In order to utilize the BESD
with the standardized mean difference effect size (Cohen’s d) it must first be transformed into its
correlational equivalent using the following formula:
𝑟=
𝑑
√4 + 𝑑 2
The correlation is divided by 2 and added to or subtracted from .50 (i.e., .50 ± r /2). Figure TN 7.1
displays the BESD for an effect size (Cohen’s d) of 1.0.
Figure TN 7.1. Binomial Effect Size Display (BESD)
Group
Expected to Pass
Expected to Fail
Treatment
.725
.275
Control
.275
.725
Taken at face value, the BESD for an effect size (Cohen’s d) of 1.0 would suggest that 72.5% of the
students in the treatment group would be expected to pass the hypothetical test. In comparison, only
27.5% of the students in the control group would be expected to pass. In other words, the BESD
indicates a success rate of 27.5% for the control group and 72.5% for the treatment group. It is important
to note that the BESD assumes a 50% base rate for both groups, creating an artificial situation to
illustrate the impact of the effect. That said, it does allow an effect size to be visualized in what some
may consider a more intuitive binomial display.
Final Report
75
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
A more conservative variation on the BESD format sets the success threshold at the median of the lower
distribution. Whenever an effect size is positive, the success rate is set to 50% for the control group. If
the effect size were negative, the success rate would be set to 50% for the treatment group. Figure TN
7.2 depicts this variation.
Figure TN 7.2. A Variation of the Binomial Effect Size Display (BESD)
Group
Above Lower
Group Median
Below Lower
Group Median
Treatment
.841
.159
Control
.500
.500
Figure TN 7.2 shows a success rate of 84.1% in the treatment group compared with a success rate of
50.0% in the control group. In other words, 84.1% of the students in the treatment group scored higher
on the hypothetical test than the control group median. In contrast, 50.0% of the students in the control
group scored higher than this value which represents a gain of 34.1 percentile points in the treatment
group. Again, using this approach the base rate is always set at 50% for the lower group.
This approach parallels the description of percentile gain above. However, instead of representing gain
(or loss) in terms of the treatment group median (i.e., the student at the 50th percentile), this approach
depicts the difference between groups in terms of the median of the group with the lower distribution.
The percentile gains presented in this report can be thought of either way. For example, an effect size
(Cohen’s d) of 1.0 can be interpreted to mean that the student at the 50th percentile in the treatment
group scored higher than 84.1% of the students in the control group, or it can mean that 84.1% of the
students in the treatment group scored higher than the student at the 50th percentile in the control group.
(For a more thorough discussion regarding interpretation of effect size, see Lipsey & Wilson, 2001;
Cooper, 2009.)
Technical Note 8: Within the context of meta-analysis, independent studies can be analyzed using a
fixed-effect or random-effects model. It is important to note that in a meta-analysis the terms fixed and
random do not have the same meaning as they do with respect to independent variables in primary
studies (see Technical Note 5). Fixed-effect models are based on an assumption of one true treatment
effect common to every study. In other words, fixed-effect models assume that exactly the same effect
size underlies all studies in the meta-analysis. Additionally, fixed-effect models are based on an
assumption that differences in observed effects are due to sampling error alone. Random-effects models
do not assume the existence of a common treatment effect. In contrast, random-effects models allow for
the possibility that the effect size varies from study to study. Stated differently, random-effects models
are based on an assumption that the true treatment effect sizes in the individual studies may differ from
one another. Random-effects models are often used to estimate this variance. (For a more thorough
discussion regarding models used in meta-analysis, see Lipsey & Wilson, 2001; Hunter & Schmidt,
2004; Cooper, 2009; Borenstein, Hedges, Higgins, & Rothstein, 2009.)
Final Report
76
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Technical Note 9: In a meta-analysis, effect sizes from multiple studies on a common topic are averaged
together to determine the effect for a specific treatment (e.g., use of Promethean ActivClassroom). Metaanalysts are interested in determining why effect sizes vary from one comparison to another. In order to
accomplish this, the meta-analyst performs a homogeneity analysis. A homogeneity analysis compares
the observed variance between effect sizes with the variance that would be expected from sampling error
alone. It provides a mechanism for determining the probability of the observed variance being attributed
to sampling error.
A homogeneous distribution of effect sizes would indicate that individual effect sizes differ from the
population mean solely due to sampling error. For some meta-analysts there would be little value in
examining study characteristics to explain observed variation in effect sizes. After all, sampling error
would be the simplest explanation for variation in effects. A heterogeneous distribution would suggest
that there are differences among effect sizes that have a source of difference (e.g., differences associated
with study characteristics) other than sampling error. Meta-analysts would then examine whether study
characteristics are systematically associated with variation in effects.
In order to test whether a set of effect sizes is homogeneous, meta-analysts calculate a statistic Hedges
and Olkin (1985) called Qt. The statistic Qt is distributed as a chi-square with one less than the number
of effect sizes as degrees of freedom. If Qt is statistically significant at the chosen level (e.g., p < .05) the
effect sizes are heterogeneous and the meta-analysts reject the null hypothesis that sampling error alone
produced the variance in effect sizes. A reasonable inference can be made that there is another source for
the difference. If Qt is not statistically significant, the effect sizes are homogeneous and the null
hypothesis that sampling error alone produced the variance in effect sizes cannot be rejected. Therefore,
sampling error alone is the most parsimonious explanation for the difference in effects.
In order to test whether a set of average effect sizes is homogeneous, meta-analysts calculate a statistic
called Qb. The statistic Qb is used to test whether the average effects from specific groupings (i.e., on a
specified moderator variable) are homogeneous. Qb is distributed as a chi-square with one less than the
number of groupings as degrees of freedom. If Qb is statistically significant at the chosen level (e.g., p <
.05) the average effect sizes are heterogeneous and the meta-analysts reject the null hypothesis that the
effect sizes were equal in the populations for each group. In other words, the grouping variable can be
considered a significant contributor to the variance in effect sizes. If Qb is not statistically significant, the
average effect sizes are homogeneous and the null hypothesis that the effect sizes were equal in the
populations for each group cannot be rejected. The grouping variable would not explain the variance in
effects beyond that associated with sampling error.
(For a detailed discussion of homogeneity analysis and the Q-statistic, see Lipsey & Wilson, 2001;
Cooper, 2009; Borenstein, Hedges, Higgins & Rothstein, 2009.)
Technical Note 10: The independent samples t-test compares the mean scores between two groups on a
given variable. The t-test is based on three assumptions: 1) the dependent variable is normally
distributed, 2) the two groups have approximately equal variance on the dependent variable, and 3) the
two groups are independent of each other. If the t-test is found to be statistically significant at the
standard alpha of .05 (p < .05) the null hypothesis that there is no difference between means can be
rejected. In other words, the two group means would be considered statistically different from each
Final Report
77
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
other. In order to avoid a violation of the second assumption, Levene’s test is used to determine if the
two groups have equal variance. If Levene’s test is found to be statistically significant at an alpha of .05
(p < .05) then the SPSS output for the t-test for unequal variances is utilized. The t-test for unequal
variances is based on a different formula and a different calculation for degrees of freedom.
Technical Note 11: In order to correct for attenuation each of the computed effect sizes in Phase I were
divided by the same attenuation factor (see Technical Note 4). Therefore, the bivariate correlations
between the Phase II variables and the observed and corrected effect sizes were identical. Consequently,
only the correlations with the corrected effect sizes were reported.
Final Report
78
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Appendix A – MRL Action Research Instructions
Promethean ActivClassroom Evaluation Study
As an MRL action research participant, you will be asked to select a specific unit of instruction, or set of
related lessons on a single topic (hereinafter referred to as unit) to be administered to two different
groups of students—an experimental group and a control group.
At the beginning of the unit, you would design and administer the same comprehensive pretest to both
experimental and control groups over the content that will be covered. If this is not a common practice,
you might want to explain to students that they are not expected to do well on the pretest since the
material has not yet been covered; it is simply a test to find out what they know prior to instruction so
that it might be determined how much they have learned as a result of instruction. Regardless of how
many items used on the pretest or the weights assigned those items, when scoring the tests, each
student’s score should be converted to a percentage.
At the end of the unit the same comprehensive posttest would be given to both groups of students. If the
pretests are not returned to students, the same comprehensive assessment used at the beginning of the
unit may be used again. If the pretests are returned to students, there should be changes made in the
posttest to ensure that students can’t simply recall the answers from the pretest. Like the pretest, the
posttest should be scored using a percentage scale.
Again, you are teaching the same unit to two different groups of students. Ideally, you would teach the
unit to both groups concurrently ensuring that the duration of the unit is the same for both groups. One
class period would serve as the experimental group and a second class period would serve as the control
group. When teaching the unit to the experimental group, you would make sure you use Promethean
ActivClassroom whenever and in ways you believe it to be applicable. When teaching the unit to the
control group, you would not use Promethean ActivClassroom.
If you are an elementary school teacher and do not have two different classes of students then you would
teach two consecutive units to the same group of students. The first unit is considered the experimental
group in that Promethean ActivClassroom is used. The second unit is considered the control group in
that Promethean ActivClassroom is not used. Please ensure that the duration of each unit is the same and
that the content of each unit is as closely related as possible without being identical or duplicated (e.g.,
complementary topics in mathematics).
If you choose to team up with a second teacher (one teacher serving as the experimental group and the
other teacher serving as the control) your unit must be designed in such a way that all facets of
instruction are as close to the same as possible (e.g., the same lesson plan, duration of lesson, class
materials, tests administered, etc.). Forms submitted to MRL need to indicate this approach so that it
may be reflected in the written report.
When the posttest is completed, each student’s pretest and posttest percentage scores should be recorded
on Form A along with general demographic information for each student. If a student does not take a
test, leave a blank space on the form to indicate a missing test. Note that the form does not ask for each
Final Report
79
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
student’s name. This has been done intentionally to comply with student privacy requirements. In short,
make sure that each row of Form A contains a specific student’s pretest percentage score (without the
percentage sign, i.e., 90% = 90), the same student’s posttest percentage score (without the percentage
sign), and the student’s general demographic information. Please use the ethnicity codes listed at the
bottom of each form when filling out the demographic information for your students. Again, do not
record student names or ID codes on the form. For the purposes of this action research project, please
refer to the following definitions when filling out the forms:
•
•
•
Free/Reduced Lunch (FRL): students who receive free or reduced-price lunches through the
National School Lunch Program (NSLP).
English Language Learner (ELL): students who have been designated as Limited English
Proficient (LEP), English as a Second Language (ESL), or Second Language Learner (SLL).
Students that have been reclassified as fluent English speakers should not be included in this
demographic category.
Special Education (Sp. Ed.): students who receive special education and related services under
the Individuals with Disabilities Education Act (IDEA).
At the end of the experimental/control units, you will be asked to collect data from students regarding
their perceptions of the unit. To do this, you should present students in both groups with the appropriate
student questionnaire (Form B—lower elementary, upper elementary, middle school, high school). This
can be done right before or after students take the posttest. In any event, this form must be completed
prior to students receiving their scores on the posttest so that those scores do not influence them when
they complete the form. You should go over the directions with students, emphasizing that responses
will not be able to be tracked back to individual students. To this end, it is best to have students
complete the form and then immediately place all forms in a large envelope that is sealed in front of
them.
The next type of data to be collected comes from you, the classroom teacher. Form C asks you to
respond to seven questions regarding your experience as a teacher and your use of Promethean
ActivClassroom. Finally, Form D asks for general information regarding your unit of instruction—
topic, duration, etc. If you chose to team up with another teacher, both teachers need to submit Forms C
and D. Please do not put both teacher ID codes on Form A. Submit them as one action research project
with the ID code for the teacher that taught each group on the appropriate scoring sheet.
It is important to note that this data will be collected and reported in a manner that provides anonymity
for teachers and students. You will be asked to provide a personal alpha-numeric ID code as opposed to
your name. Only you will know which results apply to your students. This is important, as supplying
your name removes the anonymity from the action research project. Do not use your name, or any part
of your name, as your personal ID code. As an alternative your project leader can randomly assign an ID
code based on the school or district name and a number. For example, two teachers from Some City
High School could be assigned SomeCity1 and SomeCity2 or SCHS1 and SCHS2. To maintain
anonymity, the ID codes should be determined at the start of the action research project and given to
each teacher at random to ensure that only the teachers involved know their respective ID codes.
Electronic versions of Forms A, C, and D will be given to your project leader. Please fill out the
electronic versions of these forms and print a hard copy. Doing so will help minimize data entry errors
Final Report
80
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
and allow for efficient import into statistical software for data analysis. When all forms have been filled
out (i.e., Form A with students’ pretest and posttest scores and demographics for experimental and
control groups, copies of Form B from each student, and Forms C and D filled out by you) put the hard
copies in a large envelope and turn the envelope into your project leader. Your project leader will
determine how the electronic forms will be collected (e.g., email attachment, CDROM, etc.).
After data has been collected and submitted to MRL, it will be analyzed and a report will be generated
similar to those found on MRL’s website (see http://www.marzanoresearch.com/
research/actionresearch_reports.aspx). The report will present data for each teacher listed by an assigned
numeric code. Your project leader will be provided with a list matching the assigned codes to the chosen
ID codes for your school. In this way, you might anonymously compare the pretest/posttest gains of
your students with those of other participating teachers at your school and across the evaluation study.
This data can be used to stimulate discussion regarding how to maintain expertise as professional
educators and better serve the needs of students.
In addition to these requirements, you need to videotape a single lesson (of your choosing) from the unit
for both experimental and control groups. For accurate comparison, the videotaped lesson must be the
same for both groups. The general purpose of the videotapes will be to determine the behaviors that
differentiate those teachers who obtain positive effects from Promethean ActivClassroom from those
who do not. The video camera should be set up in such a way as to capture you and your students’
interaction with the Promethean technology, perhaps off to one side of the classroom. It would be
helpful, if someone could periodically pan the classroom, to capture as much of the class as possible.
Please follow established guidelines for videotaping students set forth by your school. A sample parental
consent form has been included in this packet should you require one. If collected from parents do not
send them to MRL. MRL can analyze a variety of common digital camcorder media such as MiniDV
tape and Mini DVD. If feasible, it would be helpful if each teacher’s videotapes were combined and
transferred to a DVD-R disc (using each teacher’s personal ID code to identify his or her respective
video content) and then included in the data package sent by your project leader to MRL.
Thank you again for considering involvement in an action research project.
Final Report
81
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Appendix B – NCES Code Definitions
Figure B.1. NCES Codes and Definitions
NCES Locale Code & Category
Definitions Used by NCES
11 – City, Large
Territory inside an urbanized area and inside a principal city with
population of 250,000 or more.
12 – City, Midsize
Territory inside an urbanized area and inside a principal city with
population less than 250,000 and greater than or equal to 100,000.
13 – City, Small
Territory inside an urbanized area and inside a principal city with
population less than 100,000.
21 – Suburb, Large
Territory outside a principal city and inside an urbanized area with
population of 250,000 or more.
22 – Suburb, Midsize
Territory outside a principal city and inside an urbanized area with
population less than 250,000 and greater than or equal to 100,000.
23 – Suburb, Small
Territory outside a principal city and inside an urbanized area with
population less than 100,000.
31 – Town, Fringe
Territory inside an urban cluster that is less than or equal to 10 miles from
an urbanized area.
32 – Town, Distant
Territory inside an urban cluster that is more than 10 miles and less than or
equal to 35 miles from an urbanized area.
33 – Town, Remote
Territory inside an urban cluster that is more than 35 miles from an
urbanized area.
41 – Rural, Fringe
Census-defined rural territory that is less than or equal to 5 miles from an
urbanized area, as well as rural territory that is less than or equal to 2.5
miles from an urban cluster.
42 – Rural, Distant
Census-defined rural territory that is more than 5 miles but less than or
equal to 25 miles from an urbanized area, as well as rural territory that is
more than 2.5 miles but less than or equal to 10 miles from an urban
cluster.
43 – Rural, Remote
Census-defined rural territory that is more than 25 miles from an
urbanized area and is also more than 10 miles from an urban cluster.
Source: From NCES Common Core of Data (http://nces.ed.gov/ccd/rural_locales.asp).
Final Report
82
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Appendix C – Treatment/Control Study Data
Figure C.1. Year Two Continuation Study
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
1
-.64
1
Elementary School
Social Studies
--
--
--
--
2
-.30
4
Elementary School
Language Arts
28.00
36.00
35
5
3
1.03
3
Elementary School
Language Arts
3.00
6.00
75
4
4
-1.24
4
Elementary School
Language Arts
15.00
36.00
35
5
5
.19
3
Elementary School
Language Arts
1.50
12.00
90
4
6
-.50
3
Elementary School
Mathematics
12.00
6.00
35
3
7
.09
4
Elementary School
Mathematics
25.00
7.00
80
4
8a
.31
7
Middle School
Mathematics
18.00
6.00
40
4
8b
.77
8
Middle School
Mathematics
18.00
6.00
40
4
9
.96
8
Middle School
Language Arts
13.00
24.00
95
5
10
-.29
9
High School
Science
25.00
6.00
40
3
11
.45
1
Elementary School
Language Arts
10.00
36.00
50
5
12
-1.65
1
Elementary School
Mathematics
6.00
16.00
45
4
13
2.54
8
Middle School
Science
20.00
9.00
50
3
14
.45
8
Middle School
Mathematics
11.00
6.00
40
3
15
1.81
7
Middle School
Social Studies
1.00
12.00
50
4
16
.39
8
Middle School
Language Arts
6.00
12.00
65
3
17
.26
7
Middle School
Mathematics
10.00
24.00
80
3
Final Report
83
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
18
-1.70
9-12
High School
Foreign Languagea
18.00
30.00
98
4
19
.61
5
Elementary School
Science
2.00
12.00
70
4
20
-.15
5
Elementary School
Social Studies
3.00
12.00
20
3
21
1.11
2
Elementary School
Mathematics
2.00
24.00
90
4
22
.96
4
Elementary School
Science
10.00
.25
--
2
23
1.04
1
Elementary School
Language Arts
14.00
18.00
75
5
24
.04
2
Elementary School
Mathematics
4.00
12.00
85
4
25
.14
1
Elementary School
Language Arts
10.00
.25
50
4
26
.46
K
Elementary School
Mathematics
28.00
48.00
90
5
27
.82
5
Elementary School
Language Arts
11.00
2.00
85
4
28
1.21
5
Elementary School
Mathematics
8.00
--
90
4
29
.22
4
Elementary School
Social Studies
5.00
24.00
50
5
30
.59
K
Elementary School
Science
4.00
36.00
75
3
31
-.26
9-12
High School
Foreign Languageb
9.00
36.00
90
5
32
.76
4
Elementary School
Science
5.50
18.00
90
5
33
1.70
1
Elementary School
Science
32.00
12.00
80
3
34
.53
7
Middle School
Mathematics
7.50
12.00
90
4
35
.36
6
Middle School
Mathematics
14.00
18.00
70
4
36
1.50
5
Elementary School
Science
2.00
7.00
95
5
37
-.17
4
Elementary School
Mathematics
4.00
4.00
95
3
Final Report
84
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
38
.17
8
Middle School
Mathematics
13.00
4.00
90
3
39
-.72
2
Elementary School
Science
18.00
4.00
55
4
40
.68
10-12
High School
Mathematics
4.00
36.00
95
5
41
.14
3
Elementary School
Mathematics
16.00
36.00
85
5
42
.55
2
Elementary School
Language Arts
4.58
19.00
--
4
43
.13
--
High School
Language Arts
2.67
13.00
75
4
44
.10
--
Elementary School
Language Arts
5.67
36.00
55
5
45
-.06
3
Elementary School
Mathematics
20.58
43.00
60
4
a. Spanish. b. German.
Figure C.2. Year One Evaluation Study
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
1
.21
10-12
High School
Social Studies
10
5
20
3
2
1.22
6
Middle School
Mathematics
7
24
50
4
3
.62
6
Middle School
Social Studies
15
24
30
3
4
.06
8
Middle School
Language Arts
12
6
50
3
5
-.13
7
Middle School
Social Studies
30
24
35
2
6
.95
3
Elementary School
Language Arts
--
--
--
--
7
.12
5
Elementary School
Science
--
--
--
--
8
.48
5
Elementary School
Mathematics
27
12
85
5
9
.65
5
Elementary School
Social Studies
3
30
60
4
Final Report
85
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
10
.82
2
Elementary School
Mathematics
20
36
60
4
11
1.18
2
Elementary School
Mathematics
18
36
60
4
12
.57
1
Elementary School
Language Arts
22
10
50
3
13a
.38
2
Elementary School
Language Arts
19
4
10
2
13b
.17
1
Elementary School
Language Arts
--
--
--
--
14
-.18
8
Middle School
Mathematics
13
18
90
4
15
.57
8
Middle School
Language Arts
--
--
--
--
16
.69
3
Elementary School
Mathematics
23
48
65
4
17
.15
2
Elementary School
Mathematics
17
36
80
5
18
.45
7
Middle School
Mathematics
7
24
80
5
19
.48
6
Middle School
Mathematics
10
12
75
4
20
-.83
6
Middle School
Mathematics
11
6
60
4
21
.90
6
Middle School
Mathematics
15
3
80
3
22
.16
6
Middle School
Mathematics
8
5
75
4
23
.66
6
Middle School
Mathematics
8
2
75
3
24a
-.20
6
Middle School
Mathematics
15
1
90
3
24b
.08
6
Middle School
Mathematics
15
1
90
3
25
1.32
6
Middle School
Mathematics
11
18
75
4
26
-.13
6
Middle School
Mathematics
13
7
60
4
27
.45
1
Elementary School
Language Arts
7
12
--
3
Final Report
86
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
28
.28
8
Middle School
Mathematics
--
--
--
--
29
-.34
6
Middle School
Computer Education
--
--
--
--
30
1.34
--
Elementary School
Language Arts
--
--
--
--
31
.22
5
Elementary School
Social Studies
4
18
75
5
32
.72
5
Elementary School
Science
4
12
75
4
33
-.32
6
Middle School
Language Arts
15
6
30
2.5
34
.39
5
Elementary School
Science
16
24
70
4
35
.49
5
Elementary School
Science
--
--
--
--
36
.08
5
Elementary School
Social Studies
12
24
90
4
37
.20
3
Elementary School
Language Arts
6
5
40
4
38
.27
1
Elementary School
Language Arts
5
4
35
3
39
.03
6
Middle School
Science
9
24
80
5
40
.66
7
Middle School
Foreign Languageb
5
12
60
5
41
.83
--
Middle School
--
--
--
--
--
42
.13
12
High School
Mathematics
8
36
85
4
43a
.19
9
High School
Science
9
36
85
5
43b
.36
9
High School
Science
9
36
85
5
43c
.83
9
High School
Science
9
36
85
5
44
1.23
9-12
High School
Career Education
6
24
60
3
45
.60
9
High School
Language Arts
5
24
90
4
Final Report
87
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
46
.57
11-12
High School
Social Studies
12
36
80
4
47
.28
10-12
High School
Mathematics
9
18
80
4
48
1.01
9-12
High School
Music
15
36
40
4
49
-.06
7
Middle School
Mathematics
7
18
85
5
50
1.66
6
Middle School
Mathematics
4
12
95
5
51
1.03
3
Elementary School
Social Studies
16
2
40
3.5
52
-.38
5
Elementary School
Social Studies
32
3
40
3
53
.90
5
Elementary School
Science
13
3
45
3
54
.78
5
Elementary School
Science
25
3
80
3
55
.23
5
Elementary School
Social Studies
15
2
40
2
56
.85
3-6
Elementary School
Language Arts
11
3
75
4
57
-.55
9-12
High School
Mathematics
--
--
50
--
58
.45
9
High School
Social Studies
3
24
70
4
59
.24
--
High School
Mathematics
2
24
90
4
60
1.43
1
Elementary School
Mathematics
5
24
75
5
61
-.83
7
Middle School
Social Studies
23
4
40
4
62
-.05
2
Elementary School
Social Studies
13
27
90
4
63
1.20
5
Elementary School
Science
10
18
85
4
64
.61
10-12
High School
Science
4
12
80
5
65
-.62
11-12
High School
Mathematics
3
24
90
5
Final Report
88
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
Teacher
Effect Size
Grade Level
School Level
Content Area
# Yrs Teach
#Mos Tech
% Class Time
Confidence
66
1.54
10-11
High School
Mathematics
23
24
80
5
67
-.50
8
Middle School
Science
3
24
60
4
68
-.26
7
Middle School
Science
5
36
85
5
69a
.09
7
Middle School
Language Arts
6
54
25
4
69b
.25
7
Middle School
Language Arts
6
54
25
4
69c
-.50
7
Middle School
Language Arts
6
54
25
4
70
1.33
6
Middle School
Language Arts
13
60
80
5
71
.50
7
Middle School
Language Arts
5
36
60
5
72
.96
8
Elementary School
Social Studies
--
--
--
--
73
.15
7-8
Elementary School
Science
--
--
--
--
74
.32
9
High School
Social Studies
--
--
--
--
75
-.39
3
Elementary School
Mathematics
7
3
25
3
76
-.02
4-5
Elementary School
Mathematics
5
2
25
3
77
.14
K
Elementary School
Mathematics
14
3
20
3
78
-.24
4
Elementary School
Language Arts
25
60
50
4
79
.11
4
Elementary School
Language Arts
--
--
--
--
a. French.
Final Report
89
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010
References
Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. (2005). Comprehensive meta-analysis
(Version 2.2.050) [computer software]. Englewood, NJ: Biostat.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to metaanalysis. West Sussex, England: John Wiley & Sons.
Cohen, J. (1977). Statistical power for the behavioral sciences (Rev. ed.). New York: Academic Press.
Cooper, H. (2009). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Thousand
Oaks, CA: SAGE Publications.
Glass, G. V. (1976). Primary, secondary, and meta-analyses of research. Educational Researcher, 5, 3-8.
Harlow, L. L., Muliak, A. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests?
Mahwah, NJ: Erlbaum.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press,
Inc.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research
findings (2nd ed.). Thousand Oaks, CA: SAGE Publications.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE
Publications.
Lou, Y., Abrami, P. C., Spence, J. C., Poulsen, C., Chambers, B., & d’Apollonia, S. (1996). Within-class
grouping: A meta-analysis. Review of Educational Research, 66 (4), 423-458.
Marzano, R. J., & Haystead, M. W. (2009). Final report: An evaluation study of the effects of
Promethean ActivClassroom on student achievement. Englewood, CO: Marzano Research
Laboratory.
Murphy, K. R., & Myors, B. (2004). Statistical power analysis: A simple and general model for
traditional and modern hypothesis tests (2nd ed.). Mahwah, NJ: Erlbaum.
Osborne, J. W. (2003). Effect sizes and disattenuation of correlation and regression coefficients: Lessons
from educational psychology. Practical Assessment, Research and Evaluation, 8 (11). Retrieved
March 20, 2009, from http://PAREonline.net/getvn.asp?v=8&n=11.
SPSS. (2009). PASW® Statistics (Version 17.0.2) [computer software]. Chicago, IL: SPSS.
Wilson, D. B. (2005). Meta-analysis macros for SAS, SPSS, and Stata (Version 2005.05.23) [computer
software]. Retrieved June 12, 2009, from http://mason.gmu.edu/
~dwilsonb/ma.html.
Final Report
90
2009/2010 Second Year Promethean Evaluation Study
Marzano Research Laboratory
December 2010