Symposium 2: Data-Based Decision Making: From Policy to Effects on Teacher and
Student Learning
Abstract (max 200 words)
Effective data use is considered essential for improving educational quality in terms of
increased student achievement. However, most teachers do not use data properly, if at all.
Data use is influenced by factors in the school organization context; data and data systems
and user characteristics; moreover the policy context, i.e. pressure and support to use data in
a specific country, is also crucial for data use. In this symposium, therefore, the use of data in
several different countries is compared and contrasted. Moreover, the effects of data use on
teacher and student learning are discussed. The first paper, focusing on the policy level,
compares two countries with regard to how principals experience external expectations to use
data, for example, with regard to what goals they use data, and data (mis)use strategies. The
second paper explores factors influencing data use in five different countries at the school
level. The third and fourth paper address the effects of two data team interventions on both
teacher learning and student learning.
Objective
Data use is influenced by factors in the school organization context; data and data systems and
user characteristics; moreover the policy context, i.e. pressure and support to use data in a
specific country, is also crucial for data use. In this symposium, therefore, the use of data in
several different countries is compared and contrasted. Moreover, the effects of data use on
teacher and student learning are discussed.
Educational importance
Although some studies have shown effects of data use at the level of teacher
professionalization and student learning, results are still rare and dependent on factors at the
policy level and the level of school organization, data characteristics and user characteristics.
More knowledge on how policy and factors at the school level can influence data use and how
data use can impact both teacher learning and student achievement is, therefore, required.
Papers
Paper 1: Do differences in educational (policy) context lead to different data use? An
international comparative study.
Roos van Gasse, University of Antwerp
Policy makers around the world agree that the use of data can contribute to decision making
in schools. Data use is seen as necessary to establish a continuous process of school
improvement and as a policy instrument to hold schools accountable. Previous research
indicates that contextual factors, such as (policy) expectations to use data, can cause
differences in the way data are used in schools. On the one hand, external expectations to use
data can lead to systematic monitoring of the school quality and setting up actions for school
improvement within schools. On the other hand, an undue emphasis on data use for
accountability might undermine the use of data for school improvement. In this case, schools
might become exorbitantly accountability-oriented and therefore develop strategies and
(mis)use data to create an image of an excellent school (Schildkamp, Ehren, & Lai, 2012; Lee,
Louis, & Anderson, 2012). Given these hypothesized differences in data (mis)use due to
external (policy) expectations, the need for an international comparative study in data use
arises. We therefore compare two countries with regard to how principals experience external
expectations to use data and with regard to goals to use data, the types of data that are used
and data (mis)use strategies.
To explore the impact of the accountability and the school improvement perspective with
regard to data use, two very different educational systems will be compared: Flanders and the
Netherlands. With regard to policy expectations, there are several reasons to label the Dutch
educational context to be more accountability-oriented than the Flemish educational context.
First of all, in the Netherlands schools are (publicly) compared on the basis of central exams
whereas in Flanders central exams do not exist. Secondly, the Dutch Inspectorate is mainly
output-oriented, while the Flemish inspection uses a more process-oriented approach with
extensive school visits (De Volder, 2012). Finally, in the Netherlands the Inspectorate
publishes all kinds of facts and figures of schools (including their test scores), whereas in
Flanders this is not the case (Van Bruggen, 2010).
This paper discusses the results of a qualitative study that involves 12 semi-structured
interviews with six Flemish and six Dutch school leaders. The collected data are analyzed
through framework analysis. Due to the differences in (policy) expectations in Flanders and
the Netherlands, we investigate whether Dutch school leaders are more accountabilityoriented than Flemish school leaders. Accordingly, we analyze whether these differences in
educational (policy) context also lead to differences in the (mis)use of data. This comparative
study of two different educational systems with regard to data-use contributes to the research
field by gaining more insight in the impact of external (policy) expectations on the processes
of data use in schools and results in concrete recommendations for policy makers in their
quest to promote data-use in schools.
Paper 2: Exploring Data-Based-Decision Making in European Schools: Identifying
Enablers and Barriers
Kim Schildkamp (University of Twente), Jan Vanhoof (University of Antwerp) & Andreas
Breiter (University of Bremen, Ifib)
Some studies show that data-based decision making can lead to school improvement in terms
of increased student achievement (Campbell & Levin, 2009; Carlson, Borman, & Robinson,
2011). An important question that is currently largely unanswered, however, is what types of
data are used and how these data are being used or not used, specifically in an European
context. As studies show that there are distinct differences in the way schools use (or not use)
data, both between schools in different countries, but also between schools within one
country, a related question is which factors influence the practice of data use. This paper
addresses these questions in the context of five different countries (United Kingdom,
Germany, Poland, Lithuania, and the Netherlands).
We conducted case studies in schools of each of the five countries participating in this study, the
UK (N=4). The Netherlands (N=6), Germany (N=2), Poland (N=2) and Lithuania (N=2). The
results of our study show that schools in all five European countries have a wide range of data
available to them ranging from input, process, context, and output data. Respondents in all
countries talked about the use of data for policy development, accountability, school
improvement planning, teacher development and instructional improvement. The results of
this study confirm that organizational, data and data system, and user characterestics all
influence the use of data in the different schools in the different countries. However, in one
country or school a factor can work as an enabler (e.g. access to different types of data)
whereas in another school or in another country that same factor can be absent or work as a
barrier (e.g. there is too much data out there). Whether a factor works as an enabler or barrier
can also depend on the data user. In the case of our example, the user does not know where to
start because there is too much data available. It can also depend on the school organization.
In our example, the school could streamline the data and provide users with access to data that
are relevant to them.
What is interesting to note is that although there were huge variations in the contexts of
schools participating in this study, almost all the respondents acknowledged the importance of
data use, and also acknowledged the problems with data use in their schools. The most
common problems were problems with access to data and appropriate data systems and tools,
a lack of knowledge and skills in the use of data (and related to this a lack of professional
development and training in the use of data), and a lack of teacher collaboration. Therefore,
we believe that the way forward is to invest in professional development of teams of teachers
and school leaders in the use of data.
Paper 3: Turning Around Ourselves: Teacher Teamwork for Deep Teacher and Student
Learning
Vanessa Karwan (TIDES) & Janet Chrispeels (University of California, San Diego)
Schools in the United States serving large percentages of high needs students from diverse
backgrounds often are designated as “underperforming”, and are placed in “corrective action”.
This placement based on performance on standardized tests, has narrowed the curriculum and
student learning opportunities, undermining teachers’ professionalism and knowledge they
bring to their practice. However, some outlier schools are beating the odds. It is important to
understand the knowledge construction processes being used to shift the focus from effect
data (test results) to causal data (student work and teacher practices). The purpose of this
paper is to explore how one Turnaround school challenged their belief system through
implementing a Data Learning Loop process—reflectively looking at student work and
changing instructional practices. This knowledge sharing process is iterative and cyclical,
extending across teacher teamwork and students in classrooms. Individual teachers accrue
knowledge as they work with students in their classrooms. They bring this knowledge, tools
and artifacts to the grade level collaborative meetings, which provide a collective experience
with colleagues. After the meeting, teachers return to their own classrooms to “practice their
practice” and engage students anew the collective knowledge generated by the team. A week
later they come full circle bringing the refined individual tacit knowledge back to colleagues
to share again. The repeated process established through a Data Learning Loop facilitated
these cycles of individual and collective learning and knowledge construction. Three factors
contributed significantly to the Turnaround growth: weekly grade level collaborative
meetings devoted to looking at data, the increasing experience and sophistication in
developing student work aligned to standards with open-ended response opportunities to get
to students’ thinking, and a systematic reflective process that cycles back to students in the
classroom to guide teacher teams’ work. This study informs policymakers and practitioners
to consider the knowledge constructions necessary to shift teacher and student beliefs as
turning around a school involves a process of turning around oneself.
Paper 4: Effects of data teams
Cindy Poortman, Johanna Ebbeler, Kim Schildkamp & Adam Handelzalts (University of
Twente)
Although some evidence shows that data-based decision making can lead to increased student
achievement (Campbell & Levin, 2009; Carlson et al., 2011), most teachers need support in
the use of data. In the Netherlands, support was developed and piloted in the form of a
structured eight-step procedure for using data to improve the quality of education (see Figure
1). This study aims to answer the following research question: What are the effects of databased decision making in teams, applying the data team procedure, on teachers’ professional
development and school improvement?
We can distinguish four levels of effects for the data team procedure (Guskey, 1998; 2000;
Kirkpatrick, 1996; Desimone, 2009):
1) To what extent are teachers satisfied with the data team procedure?
2) To what extent have teachers’ knowledge, skills, beliefs and attitudes positively
changed?
3) To what extent do teachers actually use their improved knowledge and skills to apply
the data team procedure?
4) To what extent have student (learning) outcomes in relation to the data team problem
improved?
In order to be able to link professional development in terms of teacher learning and practice,
and student learning, we used a mixed methods design with a combination of a pretestposttest and an experimental-control group design. Ten data teams from secondary education
were followed during the two school years in which they were supported with the data team
procedure. The 42 schools of the same school board that did not participate in a data team,
formed our control group.
We used a combination of different instruments at the different levels of effects for
triangulation purposes. For the satisfaction level (1), we administered a short satisfaction
questionnaire to the data team members (N=55). Also, several data team participants were
interviewed (N=15). At the level of teacher learning results (2), we administered a knowledge
pretest and posttest (N=36). The interviews also concerned questions at this level.
Furthermore, we administered a ‘data use’ questionnaire as a pretest and posttest both to data
team members and their colleagues in the same school and in the control group schools
(N=1073). At the level of teachers’ use of knowledge and skills (3) we used the results of the
data use questionnaire and interview results. At the level of student outcomes (4) we used preand posttests for data regarding the data team’s problem.
The results of our qualitative and quantitative analyses show that data team participants are,
for example, satisfied about the support and the materials related to the datateam procedure.
Also, participants scored significantly higher on the knowledge posttest than on the pretest.
Finally, several teams found the causes for their problem and implemented measures to
improve student achievement. This paper further describes how the use of data in teams can
impact teacher professionalization and school improvement.
Organization of the symposium
The authors of each of the papers will highlight their key research results (15 minutes per
paper). Next, a discussant will briefly discuss the papers and pose discussion questions (10
minutes). These questions will be discussed with the audience and the audience can ask
questions themselves as well (20 minutes).
Connection to the theme
This proposal is linked to theme 1: School Effectiveness, School Improvement and School
Transformation - What do they Mean in Different Contexts and Different Paradigms. It
addresses the use of data in different contexts, and the focus is on the use of evidence for
decision making.
Word count: 1993
References
Campbell, C., & Levin, B. (2009). Using data to support educational improvement.
Educational Assessment, Evaluation and Accountability, 21(1), 47-65.
Carlson, D., Borman, G., & Robinson, M. (2011). A multistate district-level cluster
randomized trial of the impact of data-driven reform on reading and mathematics
achievement. Educational Evaluation and Policy Analysis, 33(3), 378-398.
Desimone, L. M. (2009). Improving impact studies of teacher's professional development:
Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199.
De Volder, I. (2012). Externe schoolevaluaties in Europa. Een vergelijkend onderzoek
(External evaluation in Europe a comparative study). Antwerpen: Garant.
Guskey, T. R. (1998). The age of our accountability. Journal of Staff Development, 19(4), 3644.
Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks: Corwin Press.
Kirkpatrick, D. (1996). Great ideas revisited. Techniques for evaluating training programs.
Revisiting Kirkpatrick's four-level model. Training & development, 50(1), 54-59.
Lee, M., Louis, K. S., & Anderson, S. (2012). Local education authorities and student
learning: The effect of policies and practices. School Effectiveness and School
Improvement, 23(2), 133-158.
Schildkamp, K., Ehren, M., & Lai, M. K. (2012). Editorial article for the special issue on
data-based decision making around the world: From policy to practice to results. School
Effectiveness and School Improvement, 23(2), 123-131.
Van Bruggen, J. C. (2010). Inspectorates of education in Europe; some comparative remarks
about their tasks and work. Brussels: SICI.