Educational Design Research:
appropriate to address problems
in educational practice
Tjeerd Plomp
Professor Emeritus
University of Twente
Enschede, The Netherlands
140519
ICREAMS
1
Van den Akker (1999; director of Netherlands
Institute for Curriculum Development)
about research in domain of curriculum
development and implementation:
many ‘traditional’ research approaches such as
experiments, surveys, correlational analyses,
with their emphasis on description hardly
provide prescriptions that are useful for design
and development problems in education
140519
ICREAMS
2
In this presentation:
•
•
•
•
•
•
Research functions and research designs
What is design research
Research question
Quality criteria for interventions
Generalizability in design research
New book on design research
140519
ICREAMS
3
Focus of research project
Dependent on research question/goal
research may address a need
–
–
–
–
–
to describe
to explain, understand, predict
to compare
to evaluate
to design/develop
Called: research functions
140519
ICREAMS
4
Research approaches/designs
to realize research functions e.g.:
•
•
•
•
•
Survey: to describe, compare, evaluate
Case studies: to describe, to explain
Experiments: to explain, to compare
Ethnography: to describe, to understand
Correlational research: e.g. to describe, to
explain
• Evaluation research: e.g. to determine the
effectiveness of a program
140519
ICREAMS
5
Research designs, e.g.:
Design research:
to design/develop an intervention (e.g. program,
T/L strategy, process, product, system, etc) with the
purpose
– to improve practice and contribute to body of
knowledge
OR
– to develop a theory
(and possibly: to inform decision making and policy
development)
Note: intervention is container term
140519
ICREAMS
6
What is Educational Design Research
(EDR)?
Distinction between:
•
•
Development studies aimed at research-based
solutions for complex problems in educational
practice
Validation studies aimed at development or
validation of a theory
Research on interventions versus research
through interventions (McKenney & Reeves, 2012)
Often both angles in one study!
140519
ICREAMS
7
What is Educational Design Research (EDR)?
Educational design research – type development
studies - is
the systematic analysis, design, development
and evaluation of educational interventions
with the dual aim of
• generating research-based solutions for
complex problems in educational practice, and
• advancing our knowledge about the
characteristics of these interventions and the
process of designing and developing them.
140519
ICREAMS
8
What is Educational Design
Research (EDR)?
Educational design research – type validation
studies - is
the study of educational interventions – such
as learning processes, learning environments
and the like –
with the purpose to develop and validate
theories about such processes and how these
can be designed.
140519
ICREAMS
9
Educational Design Research encompasses
systematic educational design processes.
The reverse is not true:
Not all systematic educational design can
be called research
140519
ICREAMS
10
Iterations of systematic design
cycles
PROBLEM
analysis
Revision: yes?
No? STOP
design & develop
prototype
evaluation
11
Design research methodology can be
expressed as follows:
Development of
prototype solutions
ID of problems
with practitioners
Revision
Reflection to produce
design principles
Testing solutions
in context
(adapted from Reeves, UoGeorgia, Athens, USA)
140519
ICREAMS
12
Susan McKenney, Univ of Twente:
140519
ICREAMS
13
Phases in Design Research
•
Preliminary phase:
needs and content analysis, review of state-of-art of
literature => conceptualisation & design specifications
•
Development or Prototyping phase
(iterative design phase + formative evaluation)
<=> micro-cycles of research
•
Assessment phase (semi-summative evaluation)
ALL Phases:
systematic reflection and documentation (resulting in
design principles)
140519
ICREAMS
14
On terminology:
(Educational) Design Research
stands for a ‘family’ of related research approaches
•
•
•
•
•
•
Design studies, design experiments
Design-based (implementation) research
Development research
Participative action research
Formative research
Engineering research
(e.g. Van den Akker et al., 2006):
But a number of characteristics in common!
140519
ICREAMS
15
Common characteristics
•
•
•
•
Interventionist
Iterative
Process oriented
Utility oriented: merit of design
measured
• Theory oriented: based upon &
contributing to
• Involvement of practitioners
140519
ICREAMS
16
A closer look:
Context Z:
Intervention X
Outcomes
Input => Process
Y1, Y2, …, Yn
In general:
Research goal may vary, but always:
(i) design/development of intervention
(ii) design principles or theory validation
140519
ICREAMS
17
Twofold yield in design research
Development
studies
(i) developing a research based intervention
as solution to complex problem, and
(ii) constructing (re-usable) design principles
Validation (i) designing learning environments with the
purpose
studies
(ii) to develop and validate theories about
learning, learning environments, or to validate
design principles
Implemen (i) implementing a particular program and (ii)
-tation
strategy and conditions under which
implementation can happen (design
studies
140519
ICREAMS
18
principles).
Research question in design research
If research goal is development of intervention:
What are the characteristics of an <intervention X>
to realize outcomes Y in context Z
OR in case of developing/validating a theory
(tentatively):
What is an adequate learning and teaching
strategy for <topic B> in context C in order to cope
with the main difficulties in the learning of this
topic
140519
ICREAMS
19
Examples of research questions
#1 what are the characteristics of an effective
in-service programme for mathematics
teachers through which they develop the
ability to apply student-centred pedagogical
methods (Tecle, 2003)
140519
ICREAMS
20
Examples of research questions
#2 What is an adequate learning and teaching
strategy for genetics in upper secondary
biology education in order to cope with the
main difficulties in learning and teaching
genetics, and
to promote the acquisition of a meaningful
and coherent understanding of hereditary
phenomena? ( Knippels, 2002)
140519
ICREAMS
21
Quality criteria for interventions:
1. Relevance or Content Validity:
• need for the intervention
• based on contemporary scientific insights
2. Consistency or Construct Validity
3. Practicality
- can be used in settings for which developed
4. Effectiveness:
- yields desired results
- cost-benefit ratio
#3&4: Expected versus Actual
140519
ICREAMS
22
Shift in emphasis on criteria
Stage
Criteria
Emphasis mainly on relevance
and consistency (i.e. validity),
less on practicality
Development initially: consistency and
practicality;
or
increasingly
practicality
and
Prototyping
gradually effectiveness
Preliminary
research
phase
Assessment
phase
140519
practicality and effectiveness
ICREAMS
23
Formative evaluation e.g.:
Design
specs
Global
design
Relevance
• Screening
• Focus grp
• Screening
• Screening
• Focus group
• Focus group • Focus group • Micro-eva
Consistency
• Screening
• Focus grp
• Screening
• Focus grp
Practicality
• Screening
• Screening
• Focus group • Focus grp
• Walkthrgh
• Focus grp
• Micro-eva
• Walkthrgh
• Focus grp
• Micro-eva
• Try-out
Effectiveness
• Screening
• Screening
• Focus grp
• Focus group • Focus group • Micro-eva
• Focus grp
• Micro-eva
• Try-out24
140519
ICREAMS
Partly
developed
Completely
developed
• Screening
• Focus group
• Focus group • Micro-eva
Generalizability in Design Research:
Yin (2003) about case study research:
striving to generalize findings to some broader theory:
Context Z:
Intervention X
Input => Process
Outcomes
Y1, Y2, …, Yn
design principles as intervention theory
local instruction theory
25
Generalizability in Design Research:
Yin (2003) about case study research:
striving to generalize findings to some broader theory
A theory (design principles or local theories) must be tested
through replications of findings in a 2nd, 3rd or more cases:
if same results in various cases, then ….
the theory (= design principles or local theories) might be
accepted for a much larger number of contexts.
This replication logic is the same that underlies design
research!
Analytical generalization.
(compare with ‘transferability’ in qualitative research)
140519
ICREAMS
26
Edited by Tjeerd Plomp &
Nienke Nieveen
Published by: SLO –
Netherlands Institute for
Curriculum Development.
Enschede; 2013
Part A: an Introduction
Part B: Illustrative cases
Free available from:
http://international.slo.nl/e
dr
140519
ICREAMS
27
Part B: 51 Cases of Design Research
Case Selection Tool:
•
•
•
•
•
•
Countries
Educational Sectors
Educational domains (>1)
Main aim
Educational focus
Type of intervention (>1)
See: http://international.slo.nl/edr
140519
ICREAMS
28
Part B: 51 Cases of Design Research
Case Selection Tool:
• 22 countries: CHI, SIN, IND, KOR – USA (14), NET (11)
• Educational Sectors: all
• Educ domains (>1): curriculum (16), learning & instruction
(29), subject-related pedagogy (34), school organization (3),
instructional technology (7), ICT in education (11)
• Main aim: development studies (36); validation studies
(14); implemtation studies (1)
• Educational focus
• Type of intervention (>1)
140519
ICREAMS
29
Part B: 51 Cases of Design Research
Case Selection Tool:
•
•
•
•
•
•
Countries
Educational Sectors
Educ domains (>1)
Main aim
Educational focus: teaching-learning methods; curriculum
planning; ICT in education; school management & leadership;
monitoring quality of educ; professional development;
workplace learning
Type of intervention: curr unit/course (34), educ programme
(12), learning task (9), monitoring system (4), assessment task
(2), other (4)
140519
ICREAMS
30
Thank you!
(t.plomp@utwente.nl)
140519
ICREAMS
31
Quotes illustrate that
many researchers in various ‘corners’
of our field belief that there is a need
for an alternative research approach
that is directly relevant for problems in
educational practice.
140519
ICREAMS
32
NOTE:
One resq => more than one res function
Example:
if aim of research is
to design and develop a teaching-learning strategy
for acquiring the competency of mathematical
modelling (in grade 11 & 12),
then researchers may first want to understand and
carefully describe what barriers students
experience with mathematical modelling.
But: there is primary research function
140519
ICREAMS
33
Why embark on Design Research?
Various reasons – e.g.:
1. Need to solve a complex problem in
education practice for which no how-todo guidelines are available
2. Need to elaborate and/or validate a
theory
(e.g. whether the theory of Realistic
Mathematics Education is valid in a
certain context)
140519
ICREAMS
34
Problems may exist at all levels - for
example:
System level:
Need for system for e-learning to serve specific
group of students
Institutional level:
What are effective methods for collaborative
learning
140519
ICREAMS
35
When is Design Research
Appropriate?
Eamonn Kelly (2009):
• Initial state(s) unknown or unclear
• Goal state(s) unknown or unclear
• Operators to move from initial states to goal
states are unknown or how to apply the
operators is unclear
140519
ICREAMS
36
Generalizability in Design Research:
So: design principles must be seen
as ‘heuristic’ statements providing guidance and
direction,
but do not give ‘certainties’.
When we give proper weight to local
conditions, any generalization is a working
hypothesis, not a conclusion!
(Lee Cronbach, 1975, cited by Tom Reeves)
140519
ICREAMS
37