USING EDUCATIONAL EFFECTIVENESS
RESEARCH TO IMPROVE THE QUALITY
OF EDUCATION: THE CONTRIBUTION
OF THE DYNAMIC MODEL
Leonidas Kyriakides
Department of Education,
University of Cyprus
INTRODUCTION: HISTORY OF
EDUCATIONAL EFFECTIVENESS RESEARCH
Educational Effectiveness Research (EER) addresses
the questions on what works in education and why.
Over the last two decades EER has been improved
considerably by the criticism on research design, the
sampling and statistical techniques.
Methodological advances have enabled more efficient
estimates of teacher and school differences in student
achievement to be obtained (Goldstein, 2003).
Progress was made by a more precise definition of the
concepts used and the relations between the concepts
(Scheerens & Bosker, 1997).
INTRODUCTION: HISTORY OF
EDUCATIONAL EFFECTIVENESS RESEARCH
The whole process has not contributed significantly
to the improvement of school effectiveness.
The dynamic model of educational effectiveness
attempts to define the dynamic relations between
the multiple factors found to be associated with
effectiveness.
The establishment of the dynamic model and its
empirical testing is expected to help EER to establish
stronger links with school improvement (Creemers &
Kyriakides, 2006).
PRESENTATION OUTLINE
1) The dynamic model of educational effectiveness: an
overview
2) Testing the validity of the dynamic model: Findings
and new research questions
3) Using the dynamic model to develop an EvidenceBased and Theory-Driven Approach to school
improvement
4) Using the Dynamic Model for Improvement Purposes:
Studies currently being undertaken
THE DYNAMIC MODEL OF EDUCATIONAL
EFFECTIVENESS: AN OVERVIEW
Essential characteristics of the model
Teaching and learning are dynamic processes that
are constantly adapting to changing needs and
opportunities. Effective schooling should be
treated as a dynamic, ongoing process.
The model is multilevel in nature and refers to
multiple factors of effectiveness which operate
at four levels.
The teaching and learning situation is emphasized.
THE DYNAMIC MODEL OF EDUCATIONAL
EFFECTIVENESS: AN OVERVIEW
Essential characteristics of the model
School-level factors are expected to influence
the teaching-learning situation by developing and
evaluating the school policy on teaching and the
policy on creating a learning environment at the
school.
The final level refers to the influence of the
educational system through a more formal way,
especially through developing and evaluating the
educational policy at the national/regional level.
THE DYNAMIC MODEL OF EDUCATIONAL
EFFECTIVENESS: AN OVERVIEW
Essential characteristics of the model
Factors at the school and context level have both
direct and indirect effects on student
achievement.
Some factors which operate at the same level are
related to each other.
Each factor is defined and measured by taking
into account five dimensions: frequency, focus,
stage, quality, and differentiation.
USING A MULTIDIMENSIONAL APPROACH TO
MEASURE THE FUNCTIONING OF FACTORS
Frequency refers to the quantity that an activity
associated with an effectiveness factor is present in
a system, school or classroom. This dimension may
not always be related in a linear way with student
outcomes.
Personal monitoring at school level can be
measured by taking into account how often the
principles use a monitoring system to supervise
their teachers.
The other four dimensions examine qualitative
characteristics of the functioning of the factors
and help us describe the complex nature of
educational effectiveness.
USING A MULTIDIMENSIONAL APPROACH TO
MEASURE THE FUNCTIONING OF FACTORS
Two aspects of focus are seen as important. The first one
refers to the specificity of the activities which can range
from very specific to general (e.g. a structuring task may
either refer to a part of a lesson or to the whole lesson
or even to a series of lessons).
The second aspect of this dimension addresses the
purpose for which an activity takes place. An activity may
be expected to achieve a single or multiple purposes.
If all the activities are expected to achieve a single
purpose, then the chances to achieve this purpose are
high, but the effect of the factor might be small due to
the fact that other purposes are not achieved and
synergy may not exist.
There should be a balance with respect to the two
aspects of the focus dimension.
USING A MULTIDIMENSIONAL APPROACH TO
MEASURE THE FUNCTIONING OF FACTORS
The stage at which tasks associated with a factor take place is
examined. Factors need to take place over a long period of time
to ensure that they have a continuous direct or indirect effect
on student learning.
School policy on opportunity to learn should be implemented
throughout the year and not only through specific regulations
announced at a specific point of time.
Measuring the stage dimension gives information about the
continuity of the existence of a factor but the activities
associated with the factor may not necessarily be the same.
The quality refers to properties of the specific factor itself, as
these are discussed in the literature.
Differentiation refers to the extent to which activities
associated with a factor are implemented in the same way for all
the subjects involved with it. Adaptation to specific needs of
each subject or group of subjects will increase the successful
implementation of a factor and maximize its effect on learning.
TEACHER FACTORS OF THE DYNAMIC MODEL
Teacher factors refer to teachers’ instructional
role and were found to be associated with student
outcomes (e.g., Brophy & Good, 1986; Muijs &
Reynolds, 2001; Seidel & Shavelson, 2007).
The eight classroom factors are as follows:
orientation, structuring, questioning, teachingmodelling, applications, management of time,
teacher role in making classroom a learning
environment, and classroom assessment.
The eight teacher factors do not refer only to one
approach of teaching such as the direct and active
teaching approach or the constructivist approach
but cover at least partly the main approaches to
learning and teaching.
SCHOOL FACTORS OF THE DYNAMIC MODEL
Emphasis is given to two main aspects of the school
policy which affect learning at both the level of
teachers and students: a) school policy for teaching
and b) school policy for creating a learning
environment at school.
The factors concerned with the school policy mainly
refer to the actions taken by the school to help
teachers and other stakeholders have a clear
understanding of what is expected from them to do.
Support
offered
to
teachers
and
other
stakeholders to implement the school policy is also
an aspect of these two overarching factors.
SCHOOL FACTORS OF THE DYNAMIC MODEL
The dynamic model refers to the following four
overarching school factors:
School policy for teaching and actions taken
for improving teaching practice,
Evaluation of school policy for teaching and of
actions taken to improve teaching,
Policy for creating a school learning
environment and actions taken for improving
the school learning environment,
Evaluation of the school learning environment
TESTING THE VALIDITY OF THE MODEL:
FINDINGS AND NEW RESEARCH QUESTIONS
1. A longitudinal study testing the validity of the model was
conducted (Kyriakides & Creemers, 2008).
 Participants: All grade 5 students (n=2503) from each class
(n=108) of 50 primary schools in Cyprus.
Achievement in mathematics, language, and religious education
were measured in September 2004, May 2005, and May 2006.
It was possible to provide evidence supporting the validity of
the proposed measurement framework.
The importance of using five dimensions to measure the teacher
and school factors was also identified (Creemers & Kyriakides,
2010).
The impact of school factors depends on the current situation
of the school and on the problems that it is facing.
In schools where quality of teaching is rather low, school
factors had stronger effects on student outcomes.
TESTING THE VALIDITY OF THE MODEL:
FINDINGS AND NEW RESEARCH QUESTIONS
2. A longitudinal study investigating the impact of teacher factors
on achievement of students at the end of pre-primary school
was conducted (Kyriakides & Creemers, 2009).
a.
Similarities and differences in effective teaching of two
different subjects (mathematics and Greek language) and
at two different phases of schooling (pre-primary and
primary education) were identified.
b.
The assumption that teacher factors are generic was mainly
supported but some factors were found to be more
important for one age of schooling than another.
3. A quantitative synthesis of school effectiveness studies
conducted during the last 25 years provided some support to the
validity of the model at the school level. Factors excluded from
the model were only weakly associated with student achievement
(Kyriakides, Creemers, Antoniou & Demetriou, 2010).
TESTING THE VALIDITY OF THE MODEL:
FINDINGS AND NEW RESEARCH QUESTIONS
4. We conducted a replication study in the same
50 primary schools where the original study
investigating the validity of the dynamic model
was conducted. The design of the study was
identical to that of the original study.
This study investigates one of the essential
differences of the dynamic model which has to
do with its attempt to relate changes in the
effectiveness status of schools to the changes
in the functioning of school factors.
TESTING THE VALIDITY OF THE MODEL:
FINDINGS AND NEW RESEARCH QUESTIONS
The results of DFA revealed that we can predict changes
in the effectiveness status of schools by looking at
changes in the teaching practice and changes in the
functioning of school factors (Creemers & Kyriakides, in
press).
We did not predict changes that occur in schools which
remained among the most effective.
Schools cannot remain among the most effective unless
improvement in the functioning of school factors is
observed.
Further studies to test the generalizability of these
results are needed.
Effectiveness studies should not simply try to understand
variation on the effectiveness status of schools during one
academic year but explain why specific changes in their
effectiveness status take place.
TESTING THE VALIDITY OF THE MODEL:
FINDINGS AND NEW RESEARCH QUESTIONS
To test the generalisability of the findings of these
studies by investigating
the impact of factors upon different learning outcomes
the impact of factors upon achievement of different age
group of children
To identify which factors of the dynamic model are
associated with learning outcomes irrespective of
the context and which factors have differential
effects and are therefore more relevant for policy
making in specific socio-cultural contexts.
To measure and identify the impact of system
factors
THE ESF PROJECT: MAIN AIMS
To investigate and explain differences between countries
and schools within countries in the average and differential
added value of education for different learning outcomes.
to inform policy makers about effective practices at
system, school, and classroom level contributing to the
improvement of educational quality in terms of higher
average achievement and better opportunities for
disadvantaged students.
to test the validity of the dynamic model in relation to
diversity of student intake, processes, and prospective
outcomes in order to improve the effectiveness of
education based on scientific validated models.
to elaborate on the system level factors of the dynamic
model, explore their relationships with educational
outcomes and with the school and classroom level factors,
and draw implications for policy and research.
THE ESF PROJECT: RESEARCH DESIGN AND
METHODS
In each country, a sample of at least 50 primary
schools is drawn. Tests in mathematics and science
to all grade 4 students are administered at the
beginning and at the end of school year 2010-2011.
Student level factors: We restrict ourselves to
prior-knowledge, SES, ethnicity, and gender.
These factors explain the majority of variance at
student level and can be used to search for
differential effects of classroom and school factors.
All classroom and school level factors of the model
will be measured.
THE ESF PROJECT: RESEARCH DESIGN AND
METHODS
System level factors: A description of the actors at
different layers of the system level in each country will
be conducted. Based on this analysis, we will determine
which actors have to be addressed in each country in
order to get full information about the system factors
operating in each country.
Data on system level factors will be generated through
not only a content analysis of policy documents but also by
interviews to policy makers and other stakeholders and
professionals.
A questionnaire will be developed to measure the
perceived impact of national policy on schools and will be
administered to teachers and head teachers of the school
sample in each country.
THE ESF PROJECT: RESEARCH DESIGN AND
METHODS
We will measure the perceptions of policy makers,
teachers, and school leaders about what constitutes the
wider educational environment and how this affects
policy at national and school level.
Questionnaires will be administered to policy makers,
teachers, and head teachers investigating the functioning
of factors associated with the wider educational
environment in each country.
Pilot studies will be conducted in each country in order to
test the face and the construct validity of each
instrument.
Structural equation modelling techniques will be used to
test the construct validity of the instruments.
THE ESF PROJECT: ANALYSIS OF DATA
Within-country analyses will be conducted in order to
identify the extent to which each classroom- and schoollevel factor is associated with achievement in each
outcome.
Across-countries analysis will help us identify generic
and differential factors operating at different levels.
We will search for differential impact of factors for
different groups of students in order to provide
suggestions on how education addresses the diversity in
the society.
The impact of factors in relation to equity (i.e., reducing
the gap between achievement of different groups) will
be examined.
THE ESF PROJECT: POTENTIAL IMPACT
In each country data associated with the design of the
common project will be collected.
In this way, multilevel analysis of the data across and within
each country will reveal the importance of factors of the
dynamic model.
The individual projects address topics that are relevant to
specific aspects of the theoretical framework of the
common project and provide in depth information on the
functioning of the dynamic model in different contexts.
The main aim of the ESF project is not only to generate a
theory about the dynamic perspective of effective
education but also to come up with possibilities of
establishing an evidence-based and theory-driven approach
in policy making.
USING THE DYNAMIC MODEL TO DEVELOP AN
EVIDENCE-BASED AND THEORY-DRIVEN APPROACH
TO SCHOOL IMPROVEMENT
The dynamic model of educational effectiveness could
contribute in establishing a theory-driven and
evidence-based approach to school improvement.
The model does not only refer to factors that are
important for explaining variation in student
achievement but it also attempts to explain why these
factors are important by integrating different
theoretical orientations to effectiveness (Sammons,
2009).
Teachers are offered the opportunity to use in a
flexible way this knowledge-base, adapt it to their
specific needs, and develop their own strategies for
school improvement (Teddlie & Stringfield, in press).
USING THE DYNAMIC MODEL TO DEVELOP AN
EVIDENCE-BASED AND THEORY-DRIVEN APPROACH
TO SCHOOL IMPROVEMENT
The model represents the complexity of educational
effectiveness but this representation in factors
and dimensions of factors provides opportunities to
address improvement of education in a flexible way.
By treating differentiation as a dimension of
measuring the functioning of each factor.
The model points at the possibility of defining
grouping of factors. This implies that more
comprehensive strategies with synergetic effects
can be developed in order to address the
improvement needs of each school.
USING THE DYNAMIC MODEL TO DEVELOP AN
EVIDENCE-BASED AND THEORY-DRIVEN APPROACH
TO SCHOOL IMPROVEMENT
During the last eight years, studies attempting to
develop and test the dynamic model have been
conducted and provided support to the validity of
the dynamic model (Creemers & Kyriakides, in press).
Since the dynamic model was designed in order to
establish stronger links between EER and
improvement of practice, experimental studies
and/or case studies should be conducted to identify
the extent to which schools can make use of the
proposed dynamic approach to school improvement.
USING THE DYNAMIC MODEL TO DEVELOP AN
EVIDENCE-BASED AND THEORY-DRIVEN APPROACH
TO SCHOOL IMPROVEMENT
An experimental study concerned with the use of
the dynamic model for improvement purposes at
teacher level (Antoniou, 2009) was conducted.
The teaching skills, associated with the teacher
factors of the model, can be grouped into five
levels which are discerned in a distinctive way and
move gradually from skills associated with direct
teaching to skills concerned with new teaching
approaches (Kyriakides, Creemers & Antoniou,
2009).
Teachers situated at higher levels were found to
have better student outcomes.
DEVELOPMENTAL STAGES OF TEACHING SKILLS
Levels
1.
2.
3.
4.
5.
Basic Elements of Direct Teaching
Putting aspects of quality in direct teaching
and touching on active teaching
Acquiring quality in active teaching and reaching out
Differentiation of teaching and putting aspects of
quality in new teaching.
Achieving quality and differentiation in teaching
using different approaches
LEVEL 3
1) Stage student relations
2) Stage teacher-student
relation
3)
Stage
Assessment
1) Stage Structuring
4)
Frequency
Teaching
2) Quality Application
Modelling
3) Stage Questioning
4) Frequency student relations 5) Frequency Orientation
6) Focus student relations
5) Focus Application
7) Quality: feedback
6) Stage Application
8) Focus Questioning
7) Quality of questions
9) Focus teacher-student
relation
10) Quality structuring
11) Quality Assessment
LEVEL 2
LEVEL 1
1) Frequency management
of time
2) Stage Management of time
3) Frequency structuring
4) Frequency Application
5) Frequency Assessment
6) Frequency Questioning
7) Frequency teacher-student
relation
LEVEL 5
LEVEL 4
1) Differentiation
Structuring
2) Differentiation time
management
3) Differentiation
Questioning
4) Differentiation
Application
5) Focus Assessment
6) Differentiation
Assessment
7) Stage teaching
modelling
8) Stage orientation
1) Quality teacherstudent relation
2) Quality student
relations
3) Dif teacher-student
relation
4) Differentiation
student relations
5) Focus Orientation
6) Quality Orientation
7) Differentiation
Orientation
8) Quality of teaching
modelling including
differentiation
9) Focus Teaching
Modelling
USING THE DYNAMIC MODEL TO DEVELOP AN
EVIDENCE-BASED AND THEORY-DRIVEN APPROACH
TO SCHOOL IMPROVEMENT
This study attempted to find out whether providing
teachers with training on their skills grouped in these five
levels will help them master the skills within their level
and move to the next level up (Antoniou, 2009).
We randomly split into two groups the teachers who
participated in this study and were found to be at a
certain developmental stage.
The first group was provided training by taking into
account the stage at which they were situated whereas
the holistic approach was used with the second group.
This project shows that the use of the dynamic model to
develop in-service training programmes is more effective
than the holistic approach (see Antoniou, 2010)
USING AN EVIDENCE-BASED AND THEORYDRIVEN APPROACH TO IMPROVE SCHOOL
EFFECTIVENESS: THE COUNTRY SPECIFIC
PROJECT CONDUCTED IN SLOVENIA
A) Main Aim of the Study
To identify the extent to which the
establishment of self-evaluation mechanisms
cannot in itself help schools improve their
effectiveness, but either the schools should
make use of the knowledge-base of EER provided
by the dynamic model or should first improve
their climate and then build effective
improvement mechanisms.
THE COUNTRY SPECIFIC PROJECT
CONDUCTED IN SLOVENIA: METHODS
B) Rational: Designing a multi-treatment experimental study
A sample of
achievement
students is
collaborative
50 primary schools has been selected and
in mathematics and science of grade 4
measured using the instruments of the
project (with items based on TIMSS).
The school sample will be randomly split into three
groups in order to measure the effects of three
different approaches of establishing SSE mechanisms on
student achievement in mathematics.
Different types of support will be provided to each group
of schools to establish SSE mechanisms in order to make
decisions on how to improve their policy and practice.
THE COUNTRY SPECIFIC PROJECT
CONDUCTED IN SLOVENIA: METHODS
B) Rational: Designing a multi-treatment experimental study
First group: The decision making process in establishing
criteria for SSE is left to the school stakeholders (i.e.,
students, parents, and teachers).
According to the participatory models of evaluation, the
involvement of students, parents, and teachers in defining
the criteria of SSE will eventually encourage their active
participation in using SSE for improvement reasons.
The first approach is related to the assumption that the
school stakeholders are able to develop SSE mechanisms
and define their own improvement strategies. In this way,
ownership is established and it is seen as a stimulus for
taking responsibility for school improvement.
THE COUNTRY SPECIFIC PROJECT
CONDUCTED IN SLOVENIA: METHODS
B) Rational: Designing a multi-treatment experimental study
The second treatment gives emphasis to the
establishment of a school climate that supports change.
The stakeholders of these schools will be encouraged to
express their concerns about SSE and exchange them
with each other.
At the next stage, support will be provided to the
schools to establish their own SSE mechanisms by
generating their own criteria of SSE and collecting data
in relation to these criteria.
THE COUNTRY SPECIFIC PROJECT
CONDUCTED IN SLOVENIA: METHODS
B) Rational: Designing a multi-treatment experimental study
The third group will not only be encouraged to establish
SSE mechanisms but also to take decisions for their
improvement strategies which are in line with the
assumptions of the dynamic model.
Beyond presenting the dynamic model and its assumptions
to the school stakeholders, the instruments used to test
the validity of the model at the school level will be
administered.
The results of this evaluation will be presented to the
school stakeholders and they will be encouraged to design
their own school improvement initiatives in such a way that
one of the first three priorities of their schools could be
addressed.
USING THE DYNAMIC MODEL FOR IMPROVEMENT
PURPOSES STUDIES CURRENTLY BEING UNDERTAKEN
1. A study investigating the extent to which
schools can use an evidence-based and theorydriven approach to face bullying is currently
being undertaken in five European countries.
Main aim of this study:
To find out whether a framework based on
research on bullying and on the dynamic model
can help schools to establish strategies and
actions to define and achieve goals associated
with bullying and ultimately reduce bullying.
STUDIES CURRENTLY BEING UNDERTAKEN
2. A study investigating the impact of the dynamic
approach to teacher professional development
This project investigates if the use of the dynamic model
to develop in-service training programmes is more
effective than the use of the holistic approach which is
also advocated widely for teacher professional
development.
The extent to which the impact of these two approaches
depends on whether they are offered externally or on a
school-basis is examined.
This is due to the fact that the dynamic model emphasises
the relation among school level factors (i.e., policy on
teaching and policy on the school learning environment) and
teacher professional development.
FINAL CONCLUSIONS
Projects investigating the impact of the dynamic
approach to school improvement should be conducted.
These studies will eventually reveal the strengths and
weaknesses of this approach to school improvement
and may help us identify under which conditions
teachers and schools can make use of this approach to
improve the quality of education.
These studies may contribute in the establishment of
a knowledge base of effective school improvement
efforts and encourage schools to use an evidencebased and theory-driven approach to school
improvement.
Thank you for
being attentive …