From Evaluation to Research
Description of a continuum
in the field of
Global Education
Contact: claudia.bergmueller@ewf.uni-erlangen.de
1
Overview
1.
Evaluation and Research
2.
From Evaluation to Research – a continuum
3.
Models of Evaluation in the field of Global Education and
their relevance for Research
4.
The analysis of learning effects as a challenge for
Evaluation and Research
Contact: claudia.bergmueller@ewf.uni-erlangen.de
2
Evaluation and Research
Contact: claudia.bergmueller@ewf.uni-erlangen.de
3
Research
“The activity of scientific research is to establish rates or systems
of propositions and to systematically review, in the empirical
sciences in particular hypothesis, theory, systems which are
installed and checked against experience by observation and
experiment." (Popper, 1966: 3)
Research …
1.
is a systemic activity (about a "system of records" or a "system of
knowledge");
2.
is directed towards reality (empiricism);
3.
works with the help of hypotheses, theories, and other means, and
4.
arrives at general statements ("systems theory" be established and
verified, or "causal relationships and regularities")
Contact: claudia.bergmueller@ewf.uni-erlangen.de
4
Research
Standards:
1) The results need to be provable.
2) The results need to be intersubjectively verifiable.
Contact: claudia.bergmueller@ewf.uni-erlangen.de
5
Evaluation
Evaluation is …
1.
is a systemic activity
2. is directed towards reality (empiricism);
3. works with the help of pre-determined quality standards
4. arrives at project-related statements (assessment of strengths
and weaknesses of a certain project reality)
Standards:
1)
Utility
2) Feasibility
3) Propriety
4) Accuracy
Contact: claudia.bergmueller@ewf.uni-erlangen.de
6
Evaluation and Research
Evaluation
Research
Concern
Valuation of practice based on Generation of knowledge about
verifiable criteria
a field of practice
Focus
Focus on a case
Generalization as perspective
Standards
Standards of carrying out an
evaluation on a case
Sufficient data base for the
generalization
Intersubjectivity of the
valuation needs to be
verifiable
Generation of knowledge needs
to be intersubjectively verifiable
Contact: claudia.bergmueller@ewf.uni-erlangen.de
7
From Evaluation to Research
Contact: claudia.bergmueller@ewf.uni-erlangen.de
8
From Evaluation to Research
Evaluation
From Evaluation
to Research
Research
Concern
Valuation of practice
based on verifiable
criteria
Using the knowledge from
the evaluation without
inheriting the attitude to
improve
Generation of
knowledge about a
field of practice
Focus
Focus on a case
Several similarily arranged
evaluations
Generalization as
perspective
Sensitivity for the
evaluation‘ s limitations
Sufficient data base
for the
generalization
Data collection with
methods that permit
intersubjectivity
Generation of
knowledge needs to
be intersubjectively
verifiable
Standards Standards of carrying
out an evaluation at a
case
Intersubjectivity of the
valuation needs to be
verifiable
Contact: claudia.bergmueller@ewf.uni-erlangen.de
9
Models of Evaluation
and their
relevance for Research
Contact: claudia.bergmueller@ewf.uni-erlangen.de
10
Models of Evaluation
We can distinguish three models…
1.
Concept Evaluation
2.
Performance Evaluation
3.
Impact Evaluation
Contact: claudia.bergmueller@ewf.uni-erlangen.de
11
Concept Evaluation
Examples:
1)
Evaluating the quality of an educational programme’s concept …
2)
Evaluating the quality of teaching material …
…with normatively defined quality standards/criteria
Possible aspects of concept evaluations:
-
Objectives
-
Content
-
Educational Methodology ...
Contact: claudia.bergmueller@ewf.uni-erlangen.de
12
Concept Evaluation
Possible questions:
•
Concerning the objectives: Do the educational objectives of an
activity relate adequately to the target group? Are these
objectives expressed in an understandable way?
•
Concerning the content: Does the content relate appropriately to
the objectives / the target group? Does the chosen content and its
didactic format comply with the quality criteria of Global
Education?
Contact: claudia.bergmueller@ewf.uni-erlangen.de
13
Performance Evaluation
Examples:
1)
Evaluating the implementation of a Global Education project in a school
2)
Evaluating the organisational structures of an institution
Possible aspects of performance evaluations:
-
Project management
-
Transparency
-
Educational Methodology ....
Contact: claudia.bergmueller@ewf.uni-erlangen.de
14
Performance Evaluation
Possible questions:
•
Concerning the project management: Does the allocation of
responsibilities correspond to the working conditions?
•
Concerning the educational methodology: Is the learning material
used properly / adequately for the particular target group?
Contact: claudia.bergmueller@ewf.uni-erlangen.de
15
Impact Evaluation
Examples:
1)
Evaluating the widespread impact of a certain material
2)
Evaluating the impact of an activity / project on a particular target group
Possible aspects of impact evaluations:
-
Spread
-
Satisfaction
-
Learning effects …
Contact: claudia.bergmueller@ewf.uni-erlangen.de
16
Impact Evaluation
Possible questions:
•
Concerning satisfaction: In which aspects are the stakeholders /
target groups content with the performance / conduction of the
project?
•
Concerning the learning effects: How do the target groups assess
their own attainment of knowledge? What real learning effects can
be recognized after a certain project / activity?
Contact: claudia.bergmueller@ewf.uni-erlangen.de
17
Concept Evaluation and its relevance for Research
Concept Evaluation:
comparison of concept and quality standards
Research:
comparison of concept A and quality standards
comparison of concept B and quality standards
comparison of concept C and quality standards
Sociological interest
in the correspondance
of norms of a
practical field
To which extend do the practical
concepts comply with the theoretical
discourse or the normative quality
standards/ criteria in the field of
GE/DE?
Contact: claudia.bergmueller@ewf.uni-erlangen.de
18
Performance Evaluation and its relevance for Research
Performance Evaluation: How is the project being carried out?
Has the performance been aim-oriented?
Research:
Performance Evaluation A
Performance Evaluation B
Performance Evaluation C
Interest from the
perspective of
theories on
organizational
development and
theories of project
management
Organizational Development
Project Management
Contact: claudia.bergmueller@ewf.uni-erlangen.de
19
Impact Evaluation and its relevance for Research
In the field of GE/DE 3 different types of impact can be
distinguished:
1.
the widespread impact through a project or activity, e.g.
the number of people reached, the number of material
distributed, the number and type of media coverage etc.,
2.
the impact on the institutional level, e.g. the
strengthening of a field of activity or the extension of
projects in GE/DE within a NGO,
3.
the impacts on the level of the target groups; in this
case, another distinction can be made between the
effects on the level of satisfaction statements, selfmentioned/reported learning and real learning effects.
Contact: claudia.bergmueller@ewf.uni-erlangen.de
20
Impact Evaluation and its relevance for Research
Impact Evaluation:
Widespread impact
Impact on institutional level
Self-identified/reported learning
(Cognitive) Learning effects
Research:
Impact Evaluation A
Interest from the
perspective of
communication and
media research
Interest from the
perspective of
research into teaching
and learning
Impact Evaluation B
Impact Evaluation C
distribution/acceptance
self-reflective
competencies
real learning effects
Contact: claudia.bergmueller@ewf.uni-erlangen.de
21
The analysis of learning effects as a
challenge for Evaluation and
Research
Contact: claudia.bergmueller@ewf.uni-erlangen.de
22
Analysis of learning effects as a challenge
Individual preconditions
Teacher/
Trainer/
Educator
Input
e.g.
quality
of
material
Mediators
e.g. motivation
Learning
activities
e.g. active
use of
time
Impact
Context
Contact: claudia.bergmueller@ewf.uni-erlangen.de
23
Analysis of learning effects as a challenge
1)
Learning is very complex.
There is no linearity of teaching and learning.
Learning (esp. change of human behaviour) isn’ t always a logical
response to a certain activity.
2)
Learning has different dimensions.
- Knowledge
- Competencies concerning values and judgement
- Competencies/Strategies concerning communication and
action
Contact: claudia.bergmueller@ewf.uni-erlangen.de
24
Analysis of learning effects as a challenge
Contact: claudia.bergmueller@ewf.uni-erlangen.de
25
Analysis of learning effects as a challenge
Analysing the acquisition of knowledge:
Contact: claudia.bergmueller@ewf.uni-erlangen.de
26
Analysis of learning effects as a challenge
Analysing the acquisition of competencies concerning values and
judgement:
Decision and action alternatives by giving short tales concerning
topics, which the participants have to weigh against each other or
state reason for their choice
Focus:
To what extent does an activity create the necessary cognitive
preconditions / qualifications (!) for the intended change in values and
judgement to become most probable?
Contact: claudia.bergmueller@ewf.uni-erlangen.de
27
Analysis of learning effects as a challenge
Analysing the acquisition of competencies concerning communication
and action / strategies:
Scenic challenge-situations for which future solutions have to be
found
Focus:
Examination of certain prevailing conditions like e.g.
- the capability / ability to predict future developments
- the capability / ability to set individual goals
- the capability / ability to shape change processes
Contact: claudia.bergmueller@ewf.uni-erlangen.de
28
Conclusion
We need …
… to gain experience with the synopsis of evaluations which are
carried out according to similar criteria.
… to complement evaluation with research by taking out small areas
that can be dealt with according to the standards of research.
To do so it is necessary …
… to find more adequate methods concerning different evaluation
targets
… that evaluators and researchers get into contact with each other.
Contact: claudia.bergmueller@ewf.uni-erlangen.de
29
Thank you very much!
Contact: claudia.bergmueller@ewf.uni-erlangen.de
30