Paper 1 for the EFA Global Monitoring Report, 2004
Jaap Scheerens (01/03/04)
THE CONCEPTUAL FRAMEWORK FOR MEASURING QUALITY
In this chapter an input-process-outcome-context framework is introduced as a basis
for defining quality and to categorize different measures of quality in education. It
will be shown that this framework can be used to clarify a broad range of quality
interpretations: productivity, effectiveness, efficiency, responsiveness, equity and a
more eclectic use of quality indicators. Broad applicability will also be illustrated by
addressing criticism as to the “narrowness”, ‘linearity” and authoritarian nature of
the framework. Both with respect to the assumed information requirements of
“rational planning” as regarding the control aspects of “multi-level governance”
interpretations exist that come a long way in doing justice to the complexity of social
reality. In this way some modified interpretations on both dimensions (rational
planning and multi-level control) will be described, culminating in models that could
be thought of as the complete antipodes of the more common interpretations.
In the second part of the chapter the main components of the framework will be used
as a basis for categorizing and describing input indicators, process indicators,
outcome indicators and context indicators.
An input-process-outcome-context framework for assessing educational quality
Perspectives on education quality can be clarified on the basis of a conceptual
framework that describes education. The most frequently used way to do so is to
depict education as a productive system, in which inputs are transferred into
outcomes. Steps in elaborating this basic scheme consist of:
a) including a context dimension, that functions as a source of inputs and constraints
but also as a generator of the required outputs that should be produced;
b) differentiating outcomes in direct outputs, longer term outcomes and ultimate
societal impact;
c) recognising the hierarchical nature of conditions and processes, which comes
down to considering the functioning of public education as just another example
of “multilevel governance”.
The model depicted in Figure 1 shows the basic ingredients of this framework.
1
context
inputs
Process or throughput
outputs
school level
classroom level
Figure 1: A basic systems model on the functioning of education
In the schematic presentation of the framework in Figure 1 there are various options
in choosing the levelat which level the central “black-box” is described. When
analysing the impact of policy measures at the national level one might choose the
education system in a country as the central black box. In applications where the
quality of schools is at the centre of attention one would choose the school as the level
where the transformation of inputs to outputs is studied. A perhaps more interesting
option, however, is to distinguish several levels in the central black box, for example
the national educational system, the school level and the level of the group where the
teaching and learning at school takes place, traditionally the classroom level. But
other options are possible: the students could explicitly be incorporated as a separate
level and one might want to include local community as a level as well. Finally, one
or more controlling levels could be placed in the context.
In Figure 2 the structure of the quality framework that was presented in the 2002 EFA
Monitoring Report is shown. In this model inputs are described at three different
levels, the school, the students and the household/local community. Macro-level
policies and conditions are placed in the context dimension. The central box, where
transformation processes take place, is differentiated into a school and
teaching/learning level.
2
INPUTS
School
 curriculum content
 textbooks and learning
materials
 teacher qualifications,
training, morale and
commitment
 adequate facilities
 parent/community support
PROCESS
School climate
high expectations
strong leadership
positive teacher attitudes
safe and gender-sensitive
environment
 incentives for good
results
 flexibility/autonomy




Student characteristics
 aptitude, ability
perseverance/ commitment
 nutrition and health
 school readiness
 attended ECCE
 gender




Household/community
characteristics
parental attitudes
household income
community economic and
labour market conditions
cultural/religious factors
OUTCOMES
Teaching/learning
 sufficient learning time
 active teaching methods
 integrated systems for
assessment and feedback
 appropriate class size
 appropriate use of
language
Achievement
cognitive development
 literacy, generic skills
 good citizenship
 personal development
 positive attitudes
towards learning
 healthy behaviour
Attainment
 formal completion
 diplomas/qualification
s
Standards
 official learning
objectives (desired
outcomes)
Contextual factors
 Macro-economic and fiscal
 National goals and standards for  Education system management
policies, political stability,
education, curricular guidelines,
 Participation, progression,
decentralization and governance,
sources of funding and allocation,
completion and transition
civil service quality
teacher recruitment/deployment
 Engagement and use of time
 Peer effects, parental support,
promotion policies
Figure 2: An input-process-outcome framework for assessing education quality;
source EFA Global Monitoring Report, 2002
Figure 3 is a variation on the basic scheme that was used to summarise the findings of
empirical school effectiveness research (Scheerens, 1990).
3
Context
. achievement stimulants from higher administrative levels
. development of educational consumerism
. 'covariables', such as school size, student-body composition,
school category, urban/rural
PROCESS
Inputs
. teacher
experience
. per pupil
expenditure
. parent support
School level
. degree of achievement-oriented
policy
. educational leadership
. consensus, cooperative planning
of teachers
. quality of school curricula in
terms of content covered, and
formal structure
. orderly atmosphere
. evaluative potential
Outputs
Student
achievement,
adjusted for:
. previous
achievement
. intelligence
. SES
Classroom level
. time on task (including
homework)
. structured teaching
. opportunity to learn
. high expectations of pupils'
progress
. degree of evaluation and
monitoring of pupils' progress
. reinforcement
Figure 3: An integrated model of school effectiveness, Scheerens, 1990
In this model various levels are thrown together in the context dimension, the central
black-box again has two levels, and student input characteristics are incorporated in
the outputs, namely as variables for which output indicators are adjusted. This latter
interpretation is known as the “value-added perspective” which implies that the
effects of malleable conditions of schooling over and above the influence of student
background characteristics are focussed at.
4
It is worth underlining that the incorporation of a context-dimension adds
considerably to the flexibility and broadness of applicability of the framework. As
was already indicated the context can be seen as a generator of inputs, as a level that
determines or co-determines the definition of desired outcomes that should be
generated, and as a level that judges quality and provides feedback. In more practical
terms the context-dimension gives room for situational adaptation to local conditions.
A final analytic distinction to qualify the impact of “context” is the one between
malleable conditions and “given” environmental constraints, sometimes also indicated
as “antecedent” conditions. Malleable conditions are in the hands of actors on the
scene, like national policy planners, local constituencies, school managers, and
teachers. Antecedent conditions already “exist”. Background characteristics of
students, such as cognitive aptitude or socio-economic status of their homebackground, are examples of “given” factors. At higher levels, the school or system
level, the distinction becomes more arbitrary. For example, school size could be seen
as a given condition, but also, perhaps in a longer term perspective, as a variable that
is subject to change in national policies about desired scale of educational provisions.
Another example is the composition of the student population of a school in terms of,
for example, average socio-economic status. This variable is usually treated as a
“given” condition, out of reach of policies aimed at the improvement of the primary
process of teaching and learning. However, a school might have explicit recruitment,
selection and admission policies, in order to control student composition.
Having clarified the basic “working” of the input-process-output-context framework it
can be concluded that the framework is quite general and flexible in describing the
functioning of education. This initial conclusion will be further elaborated and
critically analysed by considering the usefulness of the framework as a basis for
defining aspects of education quality and by referring to criticism and “competitors”
to the framework.
The framework as a basis for defining different perspectives on education
quality
When examining the framework that was introduced in the preceding section, at least
six ways in defining education quality can be chosen, by emphasising certain parts,
aspects or relationships from the basic framework.
a) The productivity view
According to this view, the success of the systems is seen as depending on the
attainment of the aspired outputs/outcomes. For example in the sense of a satisfactory
quantity of school-leavers that have attained a specific level (which may be
formalised as a diploma), or in terms of an acceptable level of employment of
students with a certain diploma. According to this view output/outcome/impact
5
indicators are predominant or even the only type of quality indicators that need to be
monitored.
The introduction and application of performance standards is closely related to the
productivity view of educational quality. Basically performance standards are norms,
cutting scores or “thresholds” defined on a specific output, outcome or impact
indicator. Standards can be absolute or comparative. An example of an absolute
standard would be the statement that in a particular country at least 60% of the
students at the end of lower secondary education reach a particular proficiency level
in reading literacy. A example of a comparative or relative standard would be the
statement that the countries in the European Union will all be among the 10 % highest
scoring countries in the world on a particular international achievement test in 2009.
Comparative standards are also described as benchmarks; and a specific country or
organisation to which others wish to compare themselves could then be called the
(inter)national benchmark. Standards and benchmarks in many contexts are not just
used retro-actively, in order to evaluate, but also pro-actively, in the sense of targets.
b) The instrumental effectiveness view
According to the instrumental effectiveness view there is a clear perspective for the
selection of context, input and process indicators, namely their expected effect on
outcomes. To the extent that effectiveness or production functions can be completely
specified, in other words outcomes can be totally predicted, context, input and process
indicators could replace outcome indicators. The value of certain levels and forms of
inputs and processes is determined by their instrumental potential. Clearly the
instrumental perspective offers a more dynamic handle for policy, as it considers not
only given constraints but also factors that are policy malleable. Modelling
educational effectiveness and the results of empirical educational effectiveness
research studies will be described extensively in chapter 4. The instrumental
effectiveness perspective has a particular interest in malleable characteristics at school
level. In a subsequent section of this chapter there will therefore be a specific
emphasis on process indicators on school functioning.
An important technical aspect of the instrumental effectiveness perspective is the
notion of establishing the added value of schooling. This means that the effect of
malleable input and process conditions on student performance is to be analysed over
and above the impact of student background characteristics. The most straightforward
way to accomplish this is to measure achievement at two points in time, minimally, so
that learning progress can be assessed.
c) The adaptation perspective
This view “transcends” the instrumental effectiveness perspective by not only looking
at the question how to do things right, but first of all considering the question on how
to do the right things. In other words the adaptation perspective would lead to a
critical analysis of educational goals. Conditions that allow for a continuous sounding
6
of changing contextual conditions for the education province would receive emphasis
as means, while labour market outcomes or “social capital” could be considered as
ends, according to this view. The adaptation perspective would also cover defining a
part of the school curriculum as meant to adapt to the specific environmental context
and local and national culture. As such, this perspective could be seen as an important
complimentary perspective to the instrumental effectiveness view, which takes more
or less universal standards in basic subjects for granted.
At the macro-level of national education systems the adaptation perspective could
focus at an inventory of institutional structures for continuously checking the
complicated interactions between societal demands for education and the existing
educational targets and priorities. At school level school-community relationships and
parental involvement are examples of areas that should be focused from this
perspective.
d) The equity perspective
When inputs, processes and outcomes are analysed for their equal or “fair”
distribution among participants in education with different characteristics, equity is
the primary facet of judging educational quality. Because the equity issue has several
different interpretations, these will be outlined in a bit more detail than was done for
the other quality perspective described so far.
First of all equity of achievement outcomes can be studied by considering measures of
between student and between school variation. While the primary interest of
educational effectiveness studies is to assess the impact of malleable factors over and
above student background characteristics achievement results at the individual student
level are “adjusted” for the impact of these variables. At the same time there appears
to be an additional impact of the average of relevant student background
characteristics, such as socio-economic status on achievement. To the degree that
these student level background conditions at individual or school level have a larger
impact on achievement school systems can be judged as comparatively selective
versus equitable (the recent debates about the PISA results for a country like Germany
illustrate this issue).1
Secondly, research, particularly surveys, can provide information about the extent to
which the levels of inputs and process are the same (or different) in all schools and all
provinces within countries. In other words, how equitably are the resources or
processes allocated or distributed across schools and provinces within a country?
When the teaching force in each school is described, for example, can it be said that
all pupils – in whichever school they are – have the same quality of teachers? Or the
same provision of resources, and so on? It is important to view the levels of input and
1
The German Pisa results are characterised by relatively low average achievement levels; a large
variation between students and between schools, and a relatively strong impact of SES related
background characteristics on achievement. The categorical organisation of the German school system
is seen as one of the causes of this state of affairs.
7
process provision and equity at the same time. If the achievement levels are all low
(and much lower than they should be) but there is very little variation among schools,
then we know that the schools all have the same very low level. The patterns of
variation in school provisions are relevant information for educational policy. If the
levels of school resources are very different among schools within provinces then it
could be seen as the job of the provincial authority to do something about this. If there
are large differences among provinces but few differences among schools within
provinces then the national authority must do something to ensure more equity among
provinces.
Assuming that there is some measure of the socio-economic status (SES) composition
of the student body within schools, then it would also be possible to examine if it is
the poorer SES schools that have fewer provisions and the higher SES the higher
levels of provision. Similar kinds of questions could be raised with respect to other
grouping variables of schools, like: urban/rural and private/public.
A third way of thinking of equity is based on the philosophy of compensation and
“positive discrimination”. Related, and more neutral sounding, terms in education are:
adaptive teaching and differentiation within classrooms. The basic idea is that
something extra is done for students that are in some way or another “disadvantaged”.
Examples are smaller classes, extra tutoring, ancillary services to schools, like free
school meals, adaptive teaching approaches, increased learning time. School surveys
can capture such measures or programs in a descriptive way. The distribution of
“extras” for disadvantaged learners across schools within countries could also be seen
as a specific example of equity in the sense of the first meaning, stated above
(equitable distribution of inputs).
As a fourth interpretation, a sophisticated combination of “school effectiveness” and
“equity in schooling” is addressed in studying what is known as “differential
effectiveness”. This branch of school effectiveness research specifically addresses the
question which kind of school – and instructional conditions work best for
disadvantaged as compared to more “advantaged” students.
Fifth and finally, there is the meritocracy-interpretation of equity, based on the
aspiration that each student should achieve according to his or her aptitude and
irrespective of socio-economic status. Measurement implications for assessing equity
in terms of meritocratic potential of educational provisions are the availability of
scholastic aptitude measurements or intelligence tests.
e) The efficiency perspective
This perspective can be seen as a further demand on the productivity and instrumental
effectiveness view, by considering the highest possible outcomes at the lowest
possible costs. In Table 1 the concept of efficiency of schooling is clarified by
comparing it to school effectiveness definitions.
8
Table 1: Distinction between school effectiveness and school efficiency, cited from
Cheng, 1993, p. 4
Nature of school output
In school/Just after
schooling
Short-term effects
Internal (e.g. learning
behaviour, skills obtained)
On the society level
Long term effects
External (e.g. social
mobility, earnings,
productivity)
Non-monetary
(e.g. teachers, teaching
methods, books)
School's Technical
Effectiveness
School's Societal
Effectiveness
Monetary
(e.g. cost of books, salary,
opportunity costs)
School's Technical
Efficiency
(internal economic
effectiveness)
School's Societal Efficiency
(external economic
effectiveness)
Nature of school input
It is vitally important for the economic analysis of efficiency and effectiveness that the
value of inputs and outputs can be expressed in terms of money. For determining
efficiency it is necessary that input costs like teaching materials and teachers' salaries are
known. When the outputs can also be expressed in financial terms efficiency
determination is more like a cost-benefit analysis (Lockheed, 1988, p. 4). It has to be
noted, however, that a strict implementation of the above-mentioned economic
characterisation of school effectiveness runs up against many problems.
These already start with the question of how one should define the “desired output” of a
school, even if we concentrate on the short term effects. For instance, the “production”
or returns of a secondary school can be measured by the number of pupils who
successfully pass their school-leaving diploma. The unit in which production is
measured in this way is thus the pupil having passed his or her final examination. Often,
however, one will want to establish the units of production in a finer way and will want
to look, for instance, at the grades achieved by pupils for various examination subjects.
In addition, there are all types of choices to be made with regard to the scope of
effectiveness measures. Should only performance in basic skills be studied; is the
concern also perhaps with higher cognitive processes and should not social and/or
affective returns on education be established? Other problems related to economic
analysis of schools is the difficulty in determining monetary value on inputs and
processes and the prevailing lack of clarity on how the production process operates
(precisely what procedural and technical measures are necessary to achieve maximum
output).
Relevant to the question on how useful one regards the characterising of effectiveness in
economical terms is the acceptability of the school as a metaphor for a production unit.
9
f) The disjointed view
Combinations or relations between the various elements of Figure 1 were central in
the previous views that represent a particular perspective on education quality. An
alternative view is to consider each element “on its own’ and judge whether it is
manifested in an acceptable way, or at an acceptable level. In this way one could, for
example, consider levels of teacher training, as a (minimum) requirement for being
allowed to function as a teacher, class sizes could be judged in terms of being
acceptable for being “manageable” units for teachers and students, and teaching
strategies could be rated according to norms of good practice.
The disjointed view is descriptively the simplest one, although in an evaluative sense
it is perhaps the most arbitrary one.
When considering, from a practical point of view, the way these perspectives on
educational quality are being reflected in the composition and use of actual indicator
systems it appears that the last perspective, the “disjointed view” is predominant
(Scheerens et al., 2002)
From a conceptual point of view the range of perspectives on educational quality that
can be “generated” from the basic input-process-outcome-context framework appears
to underline the heuristic value of this framework for addressing issues of defining
and measuring educational quality.
This leaves the following issues for the remainder of this chapter. The “conceptual
conclusion” about the heuristic value of the input-process-outcome-context
framework will be subjected to further critical analysis, in the next section. And
finally, more detailed information will be given about the measurement of input,
process, outcome, and context indicators.
From rational planning to growth from within; a critical analysis of the inputprocess-outcome-context framework
The input-process-outcome-context framework is a descriptive conceptual framework
that was used to categorise a range of interpretations of educational quality. In order
to critically analyse its acceptability for this purpose, it needs to be looked at in more
dynamic terms, namely in terms of its value for policies aimed at quality enhancement
in education. What are the implications of this model for the schematic picture it
offers on the dynamics of a part of social reality as complex as education? Does it
make true its claims of universal applicability or does it leave “blind spots” that
should not be overlooked? Does it lend itself as a set of tools that are useful for
different actors on the educational scene, or is it more exclusively useful for
administrators and governors? What are the limitations for its application in nonwestern cultures? Is it sufficiently amenable to the complexities of social realty? Are
there any implications for the issue of using quantitative and/or qualitative data for the
monitoring of educational quality? It is clear that, within the context of this report,
10
only a limited answer can be given to questions of these kinds that border on
important philosophical and epistemological questions. In what follows the emphasis
will be on the general question that refers to the adequacy of dynamic applications of
the framework to deal with the complexities of social reality. What will be sketched
are in fact several interpretations that vary from rather formal and almost
“mechanistic” interpretations to more interactive ones.
Rational planning as the proto-type of a dynamic application of the input-processoutcome-context framework
The key to transforming the descriptive input-process-outcome-context framework
into “something” that is more practical and action oriented is to see “outcomes” as
“desired outcomes”, goals or targets. When this step is made, it seems logical to see
inputs and processes as means that should bring about the desired outcomes. Aspects
of the context might be seen as just another category of means, or alternately as
conditions that put certain constraints on the functioning of means that are applied. Of
course the prefix “desired” also puts us in a world that is normative and evaluative.
The technical term rationality is used to describe behaviour that is goal oriented. Since
rationality lies at the basis of theories on planning and public policy-making,
decision-making theory, micro-economic theory and theories on “organisational
learning” it is rightly addressed as the rationality paradigm.
After describing the basic characteristics and “pure” forms of the rationality paradigm
modified interpretations will be sketched that progressively relax informational and
control assumptions, at the same time allowing for more uncertainty and complexity.
At the extremes of both dimensions (the information dimension and the control
dimension) mechanisms will be encountered that are the complete antipodes of the
pure rationality model, and therefore to some extent its competitors. At the end of this
section the balance will be made up with respect to the potential of various models to
improve quality in education, and implications for the monitoring of educational
quality will be drawn as well.
The pure rationality model: synoptic planning
The basic principles of the rationality paradigm are:
a) behaviour is oriented toward preferred end states (as reflected in goals or
individual well-being);
b) in situations where there is a choice between alternative ways to attain the
preferred end states, an optimal choice is made between these alternatives, which
means that profit, well-being, or other preferred end states are maximised given
the alternatives and constraints;
c) in organisational settings the alignment of individual preferences and
organisational goals is a major issue.
11
The rationality paradigm is applied in formal and less formal models of planning,
control, design and feedback and is attached to different units: organisations as a
whole, sub-groups or departments and individuals. Apart from this, procedural vs.
structural interpretations may be distinguished, the first referring to organisational
processes and the latter referring to the design (division and interconnection) of units
and sub-units.
A further important distinction has to do with the question whether goals are
considered as "given" to the social planner or designer. or that the process of choosing
particular goals is seen as part of the planning process. In the first case the approach is
"instrumental", whereas the term "substantial rationality" (Morgan, 1986, p. 37) is
sometimes used for the latter. Stated more popularly the instrumental approach is
inherent in the phrase "doing things right" whereas the substantial perspective asks the
additional question of about "doing the right things".
The ideal of "synoptic" planning is to conceptualise a broad spectrum of long-term goals
and possible means to attain these goals. Scientific knowledge about instrumental
relationships is thought to play an important role in the selection of alternatives.
Campbell's (1969) notion of "reforms as experiments" combines a rational planning
approach to social (e.g. educational) innovation with the scientific approach of (quasi-)
experimentation.
The main characteristics of synoptic planning as a prescriptive principal conducive to
effective (in the sense of productive) organisational functioning, as applied to education,
are:
- "proactive" statement of goals, careful deduction of concrete goals, operational
objectives and assessment instruments;
- decomposition of subject-matter, creating sequences in a way that intermediate and
ultimate objectives are approached systematically;
- alignment of teaching methods (design of didactical situations) to subject-matter
segments;
- monitoring of the learning progress of students, preferably by means of objective
tests.
Focussing at the primary process of schooling, the synoptic planning approach in
education applies most of all to curriculum planning, design of textbooks,
instructional design and preparation of (series of) lessons.
When the ideal of rational planning is extended to organisational structuring, related
principles about "controlled arrangements" are applied to the division of work, the
formation of units and the way supervision is given shape. "Mechanistic structure",
"scientific management" and "machine bureaucracy" are the organisational-structural
pendants of rational planning (cf. Morgan, 1986, ch. 2). The basic ideas go back to Max
Weber, who stated the principles of bureaucracy as "a form of organisation that
emphasises precision, speed, clarity, regularity, reliability, and efficiency achieved
through the creation of a fixed division of tasks, hierarchical supervision, and detailed
12
rules and regulations". Although Mintzberg's conception of the professional
bureaucracy, applicable to schools and universities, is often treated as the complete antithesis of classical bureaucracy, it should be underlined that the basic notion of
standardisation and predictability of work-processes, be it with a considerable bandwidth of individual leeway, is retained.
This “pure” form of rationality is the interpretation that is frequently criticised for
being mechanistic, linear and authoritarian. In the educational domain a “narrow”
orientation towards cognitive learning objectives is usually added to this image;
although the model is in itself “empty” as far as the selection of substantive priorities
is concerned. (One might as well follow a synoptic planning approach to stimulate
and implement affective educational goals.) Despite these criticisms it is still the
example for many approaches to educational innovation and school improvement, as
for similar applications in other sectors. The well-known logical framework model is
another case in point.
A more fundamental kind of critique addresses the information requirements that the
pure rationality model poses. In actual practice it is impossible in most situations to be
literally “synoptic” in ones goal preferences, listing of all characteristics of the
entrance situation, the full range of available alternative means, as well as having full
information on their comparative effectiveness and efficiency. Therefore several
modified forms on this dimension of information requirements have been developed;
and these will be addressed in the next section.
Modifications based on relaxing the information requirements
As modifications with respect to the information demands inherent in the pure
rationality model the following conceptualizations will be briefly sketched:
Simon’s idea of “bounded rationality”, Lindblom’s “incrementalism” (also described
as “muddling through”) and Cohen, March and Olsen’s “garbage can model of
organisational decision-making”.
Bounded rationality
Simon’s (1945) criticism with respect to the pure rationality model was aimed at the
discrepancy he saw between the information processing requirements that the model
pre-supposes and the limited possibilities that decision-makers have in real-life
situations. Where the pure rationality model demands full information about the
consequences of each and every choice, in actual practice knowledge about
consequences is mostly partial. Moreover, it is considered quite unlikely that the
preference of all possible future outcomes can be determined fully in advance.
Finally, the third requirement, namely that all alternative actions should be
considered, is considered unrealistic as well. The core idea of Simon’s bounded
rationality is that the search for alternative solutions is stopped as soon as a
13
satisfactory alternative has been found. More realistic decision-making is thus
characterised by considering an incomplete series of alternatives and possible
outcomes, and also recognising that certain alternatives may still need to be
discovered or developed, and by striving for “satisfactory” instead of “optimal”
choices. Based on this latter characteristics the term “satisficing” is coined (March &
Simon, 1958).
Bounded rationality provides a model of planning and decision-making that is less
synoptic and “mechanistic” than the pure rationality model, but it still presupposes a
clear articulation of goals and a careful search for effective means and solutions.
Incrementalism
Lindblom’s criticism of the pure rationality model is of the same nature as Simon’s,
but his alternative conceptualisation departs more radically from the pure form.
Incrementalism was explicitly developed as a “counter-model” to the pure rationality
model, on each and every aspect. This contrast is shown in the table below, based on
Hoogerwerf, 1959.
Table 2: Comparison between synoptic rational planning and incrementalism
Synoptic rational model
Clarification of norms and goals is a
prerequisite for a systematic analysis of
policy alternatives.
Policy formation is seen as an analysis of
means-goals relationships, goals are stated
first and then a systematic search for the most
effective means must take place.
The test of good policy is that it can be
shown empirically that optimal means were
selected for the appropriate goals.
The analysis (of goals and means) is
considered to be comprehensive, each and
every important factor needs to be taken into
consideration.
Analysis depends to a large extent on theory,
and scientific knowledge.
14
Incrementalism (successive limited
comparisions)
The selection of norms and goals and the
empirical analysis of alternative actions
cannot be separated and happens closely
intertwined.
Because goals and means are hard to
separate, means-goal analyses are hardly
possible or, when carried out at all, hardly
adequate.
The test of good policy is merely that
different policy-analysts can agree on a
policy, without agreeing about the most
effective means for agreed goals.
The analysis is severely limited, because
important possible effects, important
alternatives and important values are
neglected.
Analysis and policy implementation depend
on successive limited comparisons, starting
from the entrance situation, and this
characteristic severely limits the possibility
to depend on theory.
Incrementalism paints a picture of planning and decision-making that is less clear-cut
and somewhat “fuzzy”, oversees a more limited time-span, and is more in line with a
context of mutual accommodation and negotiation than a context of central
prescription and regulation.
The garbage can model
If incrementalism can be seen as a radical alternative to rational decision-making,
Cohen, March and Olsen’s “garbage can model” put matters completely upside down.
They speak of an organisation as “a collection of choices looking for problems, issues
and feelings looking for decision situations in which they might be aired, and
decision-makers looking for work” (Cohen, March & Olsen, 1972, 2). According to
these authors decision-making in what they describe as “organised anarchies” is more
like rationalising after the fact than pro-active rational planning. As a matter of fact
they see educational organisations, particularly universities, as proto-types or
organised anarchies. Such organisations, according to them, are characterised by
“problematic preferences”, “unclear technology” and “fluid participation”. Important
elements that the garbage can model brings to the fore are:
- the importance of the ideo-syncratic preferences of individual actors (and in this
there is common ground with micro-economic theory in which “non task related”,
“egoistic” preferences of decision-makers and actors in general are taken into
consideration);
- a random element in organisational decision-making;
- a specific view on the context of organisational decision-making when the
members of the organisation have a lot of leeway and autonomy.
Modifications based on relaxing the control implications
Strictly speaking the rationality paradigm does not have a control dimension. It is only
when decisional processes are “projected” in social reality and organisational
structures that an association with organisational procedure and structure is likely to
be made. The pure rationality model pre-supposes a large degree of harmony in the
organisational configuration, and this assumption might be challenged, as in fact is
done in micro-economic theory, public choice theory in particular. The assumption
that planning and decision-making follows a fixed sequence of phases, starting with
the statement of goals is challenged by a “retro-active’ orientation, in which
assessment and diagnosis come first in the planning cycle. Finally, more spontaneous
development from within, that appears to be totally uncontrolled is briefly addressed
by referring to theories on self-reference and autopoiesis.
Creating market mechanisms: alignment of individual and organisational rationality
A central assumption in the synoptic planning and bureaucracy interpretation of the
rationality paradigm is that organisations act as integrated purposeful units. Individual
15
efforts are expected to be jointly directed at the attainment of organisational goals. In the
so-called political image of organisations (Morgan, 1986, ch. 6) this assumption is
rejected, emphasising that "organisational goals may be rational for some people’s
interests, but not for others" (ibid, p. 195). The fact that educational organisations consist
of relatively autonomous professionals, and loosely coupled sub-systems is seen as a
general condition stimulating political behaviour of the members of the organisation.
In public choice theory the lack of effective control from democratically elected bodies
over public sector organisations marks these organisations as being particularly prone to
inefficient behaviour, essentially caused by the leeway that is given to managers and
officers to pursue their own goals besides serving their organisation's primary mission2.
Public choice theory provides the diagnosis of instances of organisational
ineffectiveness, such as goal displacement, over-production of services, purposefully
counter-productive behaviour, "make work" (i.e. officials creating work for each other),
hidden agendas and time and energy consuming schisms between sub-units. When
discretional leeway of subordinate units goes together with unclear technology this too
adds to the overall nourishing ground for inefficient organisational functioning; see
Cohen, March and Olsen's famous garbage can model of organisational decision-making
(Cohen et al., 1972). Not only government departments but also universities are usually
mentioned as examples of types of organisations where these phenomena are likely to
occur. Market mechanisms and "choice" are seen as the remedy against these sources of
organisational mal-functioning.
Notes of criticism that have been made with respect to the propagation of choice in
education are that parents' choices of schools are based on other than performance
criteria (Riley, 1990, p. 558), that "choice" might stimulate inequalities in education
(Hirsch, 1994) and that completely autonomous primary and secondary schools create
problems in offering a common educational level for further education (Leune, 1994).
The alleged superiority of private over public schools is the most supportive piece of
empirical evidence for the claims of public choice theory, although the significance of
the results in question is much debated (Scheerens, 1992). At the macro level there is no
convincing evidence that national educational systems with more autonomy of schools
perform better in the area of basic competencies (Meuret & Scheerens, 1995, OECD,
2001).3
Retroactive planning and the learning organisation
A less demanding type of planning than synoptic planning is the practice of using
evaluative information on organisational functioning as a basis for corrective or
improvement-oriented action. In that case planning is likely to have a more "step by
2
A more extensive treatment of the implications of public choice theory for school effectiveness
research is given elsewhere, Scheerens, 1992, ch. 2.
3
The initial results of the PISA 2000 study indicate a statistically significant effect of school autonomy
on student performance that is unadjusted for student background characteristics. When such
controls are made, as they should, the significant effect disappears.
16
step", incremental orientation, and "goals" or expectations get the function of standards
for interpreting evaluative information. The discrepancy between actual achievement
and expectations creates the dynamics that could eventually lead to more effectiveness.
In cybernetics the cycle of assessment, feedback and corrective action is one of the
central principles.
Evaluation - feedback - corrective action and learning cycles comprise of four phases:
- measurement and assessment of performance;
- evaluative interpretation based on "given" or newly created norms;
- communication or feedback of this information to units that have the capacity to take
corrective action;
- actual and sustained use (learning) of this information to improve organisational
performance.
In the concept of the learning organisation procedural and structural conditions thought
to be conducive of this type of cycles are of central importance. Examples are: the
encouragement of openness and reflectivity, recognition of the importance of exploring
different viewpoints and avoiding the defensive attitudes against bureaucratic
accountability procedures (Morgan, 1986, p. 90).
When the cybernetic principle is seen as the basic regulatory mechanism there is room
for autonomy and “self-regulation” at lower levels in the system. This is a particularly
helpful phenomenon in education systems, given the usually large degree of professional
autonomy of teachers, and tendencies to increase school autonomy as a result of
decentralisation policies.
The blind forces of autopoiesis
The core idea of Maturana and Varela’s concept of “autopoiesis”is the notion that
living systems produce themselves. Their approach is to be seen as an additional point
of view, next to the functional perspective according to which a living system
purposefully adapts to environmental conditions. The question they asked themselves
was: “what connection can a biological system make with itself as a biological
system”. Their tentative answer being that the components that make the biological
system produce themselves and each other and that the living system can only connect
and relate to itself in the form of production relations (Reneman, 2000, p. 26, 27).
How this “works” is explained by making reference to homeostatic systems.
Homeostatic systems are cybernetic systems holding a critical variable between
specific limits. “Living systems maintain their own structure between the limits
dictated by their organization” (Reneman, 2000, p. 27).
Luhmann (1995) defines complexity as a threshold where it is no longer possible to
relate every element of a system to every other element. The capacity of a system for
relating its elements depends on the elements’ connective capacity, which, in its turn
depends on the internal complexity of the elements: … “because elements must
already be constituted as complex in order to function as unity for higher levels of
system formation” (ibid, p. 24). At the same time “elements owe their unity
17
exclusively tot he complexity of the system” (ibid, p. 27). So that there appears to be a
reciprocal, circular relationship between the elements and the system as a whole.
The integrating of a system depends on selections made within the system: “Every
complex state of affairs is based on a selection of relations among its elements, which
it uses to construct and maintain itself” (ibid, p. 25). Luhmann stresses the fact that
these fundamental selections are contingent, which means that they could also have
been otherwise.
It should be noted that the implication of this view is directed internally, rather than
externally. “The system’s focus shifts from an outlook on its environment to a view of
the internal “eigen-complexity” of the system” (Reneman, 2000, p. 34).
It is beyond the scope of the present presentation to go in more detail about
Luhmann’s interpretation of autopoiesis for social systems. From the previous
citations the following core characteristics of autopoiesis in social systems can be
deduced:

repetitive, circular processes of auto-production

processes that are not planned, nor “managed” but happen unconsciously; systems
are seen as functioning without a controlling agent, or even an observing subject;

processes that are “identity seeking” (closure), but have degrees of freedom of
openness which depend on the notion that all relationships are contingent (see next
point)

processes that are “contingent”, which means that they are not inevitable “as they
are”, but could also have been different

analytically the processes that “come to be” depend on selections among an
infinite set of possible relationships

as a “starting point” for these selections Luhmann proposes what he calls the
“Sinn” or “sense” of the system; translated to organizations this could mean that
partly unconscious preferences of organization members that make up the “latent
culture” of the organization are shaped by their pre-dispositions about the meaning
of the organization and their own relationship to it.
In this way “autopoiesis” and self-reference provide a theory on the internal dynamics
of the basic, semi-conscious nature of the organization culture. One way to interpret
the “sense” of the latent culture would be to see it as the sum or the aggregate of
relevant dispositions of the organization members. It is a well-documented
phenomenon in other areas of educational research that so called “composition
effects”, like the percentage of low achieving students, or the percentage of children
from minority backgrounds, are quite important in explaining various aspects of the
functioning of a school, the achievement results among them. Form a recent German
study the researchers concluded that the nature of a school changed after the
percentage of minority children reached a level of 40%. Similar reasoning could be
applied to the compositional effects of teacher characteristics.
18
According to the image of autopoiesis and self-reference in organizations “control
from outside” has been replaced by “development from within” and “culture” is
brought to the fore as a perspective to analyze the functioning of schooling. As far as
the effectiveness of the functioning or educational organizations is concerned, the
importance of the individual value-positions of the staff members of the school and
their aggregates (a teacher compositional effect) are stressed. In its turn this points at
the importance of selection policies, of teachers, students and school heads. As far as
the internal dynamics of school functioning are concerned the concept of autopoiesis
underlines the importance of cyclic processes that go well or go bad. The “good” in
this case would be organizational learning, and learning from ones mistakes. The bad
would be getting stuck in repeating failure, accumulation of interpersonal distrust and
uni-lateral actions. Argyris and Schön (1974) provide many examples of what they
describe as obstacles to organizational learning. Studying such complex internal
processes would require qualitative research, like participant observation, open
interviews and focus group techniques.
The merits of the input-process-outcome-context framework for analyzing educational
quality revisited
The framework allows for a broad range of quality interpretations: productivity,
effectiveness, efficiency, equity and responsiveness. All of these quality perspectives
are defined in terms of specific relationships between the components. The “eclectic”
interpretation values each of the components “on its own”. The model is not about
prescribing specific substantive educational objectives; in this the model is just an
“empty” descriptive tool that can be used to systematically approach all kinds of
educational priorities. The presentation, particularly the part on “informational”
modifications of the pure rationality model, clearly shows that there is no ground to
limit application to “closed”, “mechanistic” or “technocratic” interpretations. In most
practical situations there is simply too much complexity and “contingency” because
of situational constraints for authoritative solutions to be feasible. Instead, to the
extent that scientific information is available, it should best be seen as input that has
to be further verified in interactive processes of actors in the local setting. All that was
said, (and will be said in the forthcoming sections) on the importance of context, not
just the institutional but also the cultural context, precludes such mechanistic
interpretations. Subsequent modifications of the model with respect to the control
dimension do away with autocratic images and underline the limits of social
engineering and centralist control. Autonomy at the base of the system, in our case
that means autonomy of teachers in classrooms, has always been a characteristic of
educational organizations. Various modified interpretations of the rationality model
sketch the possible advantages, but also the risks, of further stimulating this
autonomy. The risks being spelled out by, for example, public choice theory, the
“garbage can model” and by the concept of self-reference in organizations. The
19
concept of functional decentralization, which implies that centralist control in some
domains, may be combined with decentralization in other domains, might allow
considerable autonomy at various lower levels of the system, but still keep matters
checked on one or two important dimensions, outcomes for example.
As far as the monitoring of educational quality is concerned the various
interpretations of quality and the modifications of the rational model would call for a
broad set of methods, including methods that do justice to the interactive nature of
some of the basic processes in education.
The framework that was introduced, and that in the remaining part of this chapter will
be used for the heuristic purpose of generating indicators to measure educational
quality, includes political processes, and negotiation, but does not in itself take a
political position. It will therefore not satisfy those who opt for an approach to social
analysis that starts from a particular political a priori, for example what was called an
emancipatory approach in the seventies.
Measuring educational outcomes
In this section outcome indicators will be discussed. Outcome indicators are central in
productivity and effectiveness interpretations of educational quality; but also play an
indispensable role in assessing the equity, efficiency and responsiveness of schooling.
A distinction is made between output, outcome and impact indicators. Output
indicators are seen as the more direct outcomes of schooling, and most likely
measured by means of a form of student assessment, like a standardized achievement
test. Attainment indicators, as for example the number of students that complete a
certain period of schooling without delay, are of a more administrative nature. Impact
indicators are indicators on the social status of students that have reached certain
levels of educational attainment.
achievement and attainment measures
Educational indicators based on output (achievement) and outcome (attainment)
indicators have a central place in the education quality debate. Considering output and
outcome indicators as the core criteria to judge educational quality confirms to the socalled productivity view on education, which is strongly rooted in economic
perspectives. In the educational province the term “effectiveness” is often used to
express a similar focus on outputs and outcomes. Yet, there are important, yet
gradual, differences between the economical/sociological orientation towards
productivity and the educational one. The prime focus in economic and sociological
orientation toward educational productivity is the prediction of societal, economic
well being at individual and organisational level. The preoccupation of psychological
and educational views on productivity are more “forward looking” in the sense that
the emphasis is on finding malleable conditions that can maximise educational output.
20
The fact that this dualism is not at all watertight is illustrated by the micro-economical
approach of “education production functions”, which is similarly oriented to finding
malleable conditions at school and classroom level. One could schematically
represent these two orientations as two kinds of causal relationships:
-
malleable educational conditions and output;
school outcomes and their societal impact.
It is also possible to depict this as one, more complicated, causal chain:
Malleable educational conditions - output/outcomes – societal impact
The focus in this section will be on the distinction between two kinds of
“productivity” or “effect” indicators, “output” indicators as measured by a certain
kind of student achievement measure, on the one hand, and “outcome” indicators
measured by means of educational attainment measures. It will be argued that student
achievement based indicators have a more logical connection with the educational
perspective on productivity, while attainment indicators fit better with the
economic/sociological perspective. Yet, these functions are also, to some extent
interchangeable, and the more recent interest in competencies and key-competencies
may take away the edge of this contrast completely. This means that competencies
can very well be seen as the “dependent” variables in “internal” school productivity
models, and as the “independent” variables in “external” models on the societal
impact of schooling.
The educational perspective
The predominant criterion in school effectiveness studies from various disciplinary
origins is achievement. Attainment measures depend on formal levels in the school
careers of pupils. Roughly speaking educational attainment scores express the level
that individuals or groups of pupils have reached after a certain number of years of
schooling. Examples of discrete attainment levels are the end of the primary school
period and the end of the secondary school period. Particularly when an educational
system consists of many school types, to which societal value is attributed in various
degrees, attainment scales can become quite differentiated (see e.g., Bosker & Van
der Velden, 1989).
When discussing the option of either choosing attainment measures or achievement
measures, various underlying dimensions for this choice can be discerned. First of all
the choice may depend on different connotations of effectiveness (e.g., maximisation
of output i.e. the quantity of educational attainment, vs. enhancing quality). Secondly,
preferences concerning band-width vs. specificity of output measures could determine
the choice, i.e., the question whether an overall output measure or a more narrowly
defined performance indicator is to be preferred. Thirdly, the question of the
21
predominance of a more practical vs. a scientific interest in establishing effectiveness
may lie at the background of this choice.
Attainment measures are close to the economic notion of productivity as
maximisation of output, where output is measured as the amount of products that
results from a particular production process of schooling. Achievement, on the other
hand, fits better in an interpretation of effectiveness in terms of quality. Achievement
tests as effectiveness criteria capitalise on more fine-grained quality differences of the
units of output.
Attainment measures are cruder output measures than achievement tests, but at the
same time they usually imply a broader coverage of the whole spectrum of
educational objectives. The passing of a final examination (attainment indicator)
depends on achievement in many subjects, whereas achievement tests in school
effectiveness studies are often limited to arithmetic and language tests.
School effectiveness is both a subject of scientific inquiry and an applied field of
interest in educational policy and management. When issues of consumer demands,
monitoring of schools and accountability are at stake one cannot do without
attainment indicators. In the case of inquiry into determinants of school effectiveness,
i.e., input-output, or input-process-output studies in order to define school
improvement strategies, output indicators are required that differentiate more strongly
between qualities of the units of output and achievement tests are likely to be chosen.
In summary, attainment measures are called for when purely economic and applied
perspectives of productivity and effectiveness predominate or in case one wishes to
explore, in the tradition of sociology of education, the contribution of schools to a
person’s status attainment. Achievement measures are more likely to be chosen when
quality of education is at stake and when a more psychological interest in cognitive
development (or an educational interest in schools as organisations) has the upper
hand.
Finally, it should be mentioned that the choice between attainment or achievement
can be avoided in two ways: (a) by using both, and (b) in the case of a decision
oriented use of achievement tests (as when performance standards in the form of
cutting scores on tests determine further career options).
Intermediate and 'ultimate' effect measures
Are attainment or achievement measures obtained at the end of a particular period of
schooling to be considered as the ultimate productivity measures or would only more
long term civil effects of schooling, such as employment or job level reached by
graduates, qualify as such? Or, moving into the other direction on the scale of
“ultimateness” of effectiveness measures, could we use intermediate effects like
attendance and drop-out rates as substitute effectiveness criteria? (e.g., Rutter,
Maughan, Mortimore, & Ouston, 1979).
Searching for ultimate school effects is like looking for the holy grail, since one can
go on and on in stating even more ultimate effects. The most likely, be it arbitrary,
22
points in the school careers of pupils to measure school effects are indeed when a
particular period of schooling is terminated and transition to a higher school type or
into the labour market takes place. Post-school effect measures could be seen as
important in macro-level applications of educational indicators for purposes of
monitoring national school-systems. Also, post-school effect-measures could be seen
as important criteria to gain insight into the predictive validity of effect measures at
the end of the period of schooling. Attendance and drop-out rates are better treated as
process-measures in school-effectiveness studies, because they generally function as
means rather than as desired ends of schooling.
General vs. curriculum specific achievement tests
When the decision is taken to use achievement rather than attainment output data,
there is a further option in the choice of the type of test. Madaus, Kellaghan, Rakow,
and King (1979) have provided arguments in favour of curriculum-specific tests and
against the use of general achievement tests (like the Scholastic Aptitude Test). One
of their arguments is that larger school (or class) effects are demonstrated when
curriculum specific tests (exams in their case) are used. Before offering a few lines of
thinking in determining the choice of output measure along this particular dimension,
it should be remarked that general vs. curriculum specific achievement measures
should be seen as a continuum with many discrete scale-points rather than a
dichotomous choice between two extremes.
Varying from curriculum specific to general aptitude measures one could discern the
following types of measures:
- trained test items;
- content specific measures;
- Rasch-scales of narrow content areas;
- subject specific tests;
- general scholastic aptitude tests;
- intelligence tests.
A general guideline to choose from these alternatives would be to use the more
specific measures up to the degree that the application purpose is closer to the micro
situation of classroom-instruction. A line of thinking which perhaps offers a more
fundamental solution to this problem of choice, would be to choose the type of
outcome measure that has the greatest predictive validity with respect to the more
ultimate educational effects. To give an example: when measuring achievement at the
end of a specific stream of vocational education, we might prefer content-specific
measures, assuming a close connection between the curriculum and skills that are
required in the job-situation. It should be noted here, however, that this latter kind of
criterion choice further depends on the theory one holds about the relationship
between education and the labour market. Departing from a credentials or screening
theory, certification itself would be the best criterion for effectiveness, whereas
achievement is more connected to the human capital philosophy.
23
Controlling for confounding of measures of effect
So far it was assumed that the measures on attainment or achievement scales can
indeed be interpreted unequivocally as criteria of productivity or effectiveness. This is
not so. One source of confounding of these output measures that is well-known is the
initial or even innate ability of pupils. In school effectiveness studies it should be
attempted to separate the contribution of innate characteristics of pupils from the neteffect of school characteristics. Although this source of confounding the interpretation
of output measures is well-known, critical reviewers of effective schools research
have noted that in many cases control for innate characteristics of pupils is inadequate
(Purkey & Smith, 1983).
A second source of possibly biased interpretations of school effects are selective
policy- measures at the school level. Examples are: lenient vs. a strict policy in letting
pupils pass from one grade to the next, a more or less conservative policy in allowing
students in secondary education to go in for their final examination, a more or less
reluctant attitude when it is to be decided to send pupils to special schools (primary
education) or to lower categories of secondary education (when a country has a
differentiated secondary education system). This type of selectivity bias is usually
accounted for when economic measures of school output are used, for instance,
number of graduates divided by the total number of students in the cohort that entered
the school as many years back as the normal duration of the school-period. Usually in
individual level measures of attainment selectivity will also be accounted for, because
drop-outs will still receive a score on the attainment scale. When achievement data
aggregated to the school level are used as the effect-criterion this type of selectivity
bias is usually neglected, however. The consequence of this practice is that the
corresponding estimation of school effects might well be confounded and
corresponding policy- or managerial decisions consequently unfair. The general
solution to this problem would be to obtain some kind of measure of the selectivity
policy of schools and use this information as an additional independent variable
(defined at the school level) to separate its effect from the other independent variables
that stand for the more genuine determinants of school effectiveness. Statistical
techniques to model attrition bias may help in solving the problem of separating
selectivity effects from the influence of other independent variables (see e.g.,
Hausman & Wise, 1979).
Although generally people use the school effect concept without any hesitation, there
are, however, many different underlying operationalisations. Some, of course, have to do
with the subject matter areas in which the performance or ability of the pupils was
assessed (see also Chapter 3), others have to do with gross versus value added indicators.
More important is that when using value added measures different corrections are being
made before the actual school effects are assessed, i.e. in different research studies
different covariates are used. When looking at the type of covariates used, four different
approaches can be distinguished, which to be indicated as “gross school effects“,
24
“unpredicted achievement based school effects”, “learning gain based school effects”,
and “unpredicted learning gain based school effects”.
Gross school effects
The first operational definition uses as the measure for school effect the mean
(uncorrected) achievement score of pupils in a certain school. The value of this
definition lies in its use within a criterion referenced framework: if a standard is set (or:
if a growth continuum is specified) a priori this gross school outcome measure provides
the information to judge whether the school performs above, at or below the standard. It
does, however, not imply that all pupils within that school meet the standard. This
definition can be labelled as the 'gross' school effect. In operational terms it is the mean
achievement, i.e. it is averaged over the pupils within a school, possibly with a
correction for sampling error.
School effects based on the assessment of over- or under-achievement as compared to
predicted achievement
The second operational definition starts from predicted achievement. A prediction
equation is estimated from the data, where achievement is predicted from aptitude,
socio-economic status, age, gender, ethnicity-status, and other student variables. Next the
parts of the distribution of student achievement scores that are above and below the
predicted level are estimated. The reasoning behind this approach is that schools widely
differ in their student populations, and that, since these background variables have a
strong relationship with achievement, their effects on achievement should be partialled
out. Most of these variables are static, and not subject to -much- change, though the
aptitude of a child may. For this reason the aptitude assessment should - ideally - take
place before or at school entrance.
School effects based on learning gain
The third operational definition can be seen as a specific case of the second one:
achievement is predicted from prior achievement or -what is more appropriate- the
difference between both is used. Once again the same argument applies as in the case of
the aptitude assessment. If prior achievement is assessed at a later point in time than
school entrance, the school effect transforms into the effect of a school on its pupils
within a certain time-interval.
School effects based on predicted learning gain
The last and seemingly most strict definition combines the previous two: predicted
learning gain forms the basis. Using a post-test score corrected for a pre-assessment
score, this score is in its turn corrected for aptitude, socio-economic status, age, gender,
ethnicity-status, and other student variables, since these are related to the learning
progress that pupils make. In this case prior achievement as well as aptitude should ideally - be assessed at school entrance.
25
In the literature on school effectiveness school effects are mostly referred to as value
added measures, i.e. some kind of intake adjustment is applied. We have seen however,
that in actual fact there are at least three different sets of value added based school
effects. Except for the first definition of a school effect (gross school effect), where no
adjustment for intake differences between schools is employed, the value added based
definitions can be made more strict by not only correcting for the student level effect of
the covariates but also for the potential extra effects of their aggregates. It is often
observed that, for example, the average socio-economic status of the student populations
of schools has an effect over and above the individual socio-economic status variable.
This indicates that being a working class child has a negative effect on achievement, but
being also in a school with a majority of working class children has a substantial extra
negative effect on achievement as well.
In all cases, however, causal attribution (showing that schools cause the effect) is
difficult. Raudenbush & Willms, argue that, since randomisation cannot be achieved,
assignment of students and schools to treatments should be "strongly ignorable". This
qualification expresses that the different treatment outcomes for a student and a
school "are conditionally independent of treatment assignment given a set of
covariates" (Raudenbush & Willms, 1995, p. 312) which implies that value added
measures should be preferred. But on the other hand, it is possible that value added
based school effects underestimate true school effects. When "privileged" students are
more likely to choose "good practice" schools, controlling for student's socioeconomic status, for instance, results in over-adjustment. To complicate things
further, it should be pointed out, that in compensatory systems, school effects may
appear only after adjustments for covariates are made. In compensatory systems
unfavourable scores on the covariates are compensated by (and thus correlated with)
more intensive and adapted educational practices. And moreover, these school effects
then can disappear again after context variables, that are negatively associated with
good educational practices for the same reason, have been taken into account.
A note on the size of school effects
The question referring to school effectiveness and the size of school effects should be
distinguished from questions about the practical relevance of school effects.
There are at least two kinds of practical issues at stake when dealing with this latter
question. First of all one should consider the situation where information on the value
added to educational outcomes by each school is used for political or practical
decision making. Examples are: rewarding well-performing schools by providing
them with extra resources, and comparing schools among each other in so called
league tables. The purpose of these league tables being the provision of information to
parents to facilitate the choice of a school for their children. Goldstein and Thomas
(1996) have shown that, at least in the UK, it is possible to discriminate significantly
between schools only when extremely high and extremely low schools are compared.
Goldstein (1996) provides an example in which "the confidence intervals for each
26
school cover a large part of the total range of the estimates themselves, with three
quarters of the intervals overlapping the population mean ..." (p. 5). The implication
being that it is generally not possible to discriminate in a precise and reliable way
between schools for the practical concerns in question. A second type of practical
implication of "value-added" school effects is by considering the consequences for
individual pupils. It has been established repeatedly in the literature (Purkey & Smith,
1983; Brandsma & Doolaard, 1996) that relatively small differences between schools
will nevertheless have important consequences for the educational career of individual
pupils. Purkey and Smith, for instance, discerned an entire school year's difference
between the average pupil in the most effective schools and the average pupil in the
least effective school.
Brandsma and Doolaard demonstrate that pupils from effective schools have better
chances to be advised to pass on to "higher" types of secondary education in the Dutch
educational system. These authors distinguish between four IQ-bands, and show that in
highly effective schools pupils from the three lower bands score at a level that is
equivalent to two curricular tracks (out of a total of 5) higher than pupils from ineffective
schools. In the highest IQ-band the difference is equivalent to one track.
A point that is made by Bosker and Witziers (1996) in this context is that school effects,
small as they may be, have implications as described in the above, for not just one, but
for all pupils of the cohort the effects were measured for.
The economic perspective: theories and research findings
The economist approach to education generally analyses the rational choice behaviour
of actors in education. Particularly the question about the so-called utility of a person
who follows education is studied. This means that all the costs a person has to make
to follow a particular education program are counted on the one hand. On the other
hand the benefits in terms of increased earnings (i.e. the difference in earnings
between having obtained a particular qualification and not having obtained that
qualification) are calculated. This is the basis of the best-known economic theory on
education: human capital theory. Other theories also consider the utility of employers
to engage persons with certain qualification levels. The research that economic
theories have generated gets particularly interesting for the topic at hand, when the
question is raised about the competencies that make qualified individuals more
attractive to companies than unqualified individuals. In their turn these competencies
could be taken as key targets for schools and training institutes to maximise the
external relevance of their programs.
Human capital theory treats education as a trade-off between costs and benefits. “ In
this theory, individuals incur costs, including tuition and foregone earnings while in
school, to acquire skills that increase their productivity and result in subsequent wages
higher than they otherwise would have earned. A long tradition of research has
27
focussed on the magnitude of the return to educational investment, showing that
human capital investments tend to pay off at least as well as investment in physical
capital” (Levy and Murnane, 2001, p. 154). One of the research supported
characteristics that was found in explaining this general result was that employees
with more formal education have better ability to deal with unanticipated events (like
non-routine problems at work or forced job changes), than do workers with little
formal education (ibid, p. 154).
Signalling theory concentrates more on the perspective of the employers. Employers
use the information on formal levels of schooling as a “signal” that potential
employees possess certain skills that are valued for the work in question. Recent
research, the International Adult Literacy Study has shown that it is not just the
formal qualification level that counts, but some competencies, in this case literacy,
has an effect over and above the formal level of schooling.
Principal agent theory presents an economic perspective on the utility functions of
employees on the one hand and managers on the other. The theory assumes that the
agent (the employee) has a utility function that may differ from what is valued by the
employer. For example, a certain degree of leisure, that employees could permit
themselves may be pleasant for the former but not so much for the principal
(employer). In theory the employer would need to closely monitor the behaviour of
the employee to minimize such non task-related priorities or activities. Since
monitoring may be difficult and costly, relying on employees having acquired certain
basic skills and attitudes that give some guarantee of reliability, education functions
as a substitute for monitoring and supervision. It is interesting to note that the
competencies in question are not just technical skills but also more attitudinal
characteristics like perseverance and honesty.
Levy and Murnane (2001) summarize research findings that are the result of various
kinds of economic research that are driven by these theories and have generally been
targeted to the question what competencies, related to formal schooling, could explain
the attractiveness of qualified individuals to employers. They mention five “key
competencies”:
1) Basic reading and mathematical skills are important in determining long-run
labour market outcomes, including the ability to adjust to changing circumstances.
2) The ability to communicate effectively, both orally and in writing, is important in
determining long-run labour market outcomes.
3) In modern firms it is increasingly important to possess the ability to work
productively in groups.
4) The latter condition also emphasises elements of “emotional intelligence”
including the ability to relate well to other people.
5) Familiarity with computers is of growing importance in the labour market.
28
They present their findings as tentative, however. The fact that employers are ready to
pay wage premiums to employees with higher levels of formal schooling may be
explained by the hypothesis that these employees have better communication and
teamwork skills and or are more apt to adaptation and continuous learning, but the
evidence is inconclusive. These authors also point at a number of institutional
arrangements that blur the picture of a straightforward relationship between
companies hiring and promotion policies and acquired key-competencies. They
mention “the organization of work, the links between firms and educational
institutions, and the nation’s distribution of income as modified by the welfare state”
(ibid, p. 167).
Identifying key competencies as a general approach to closing the gap between
schooling and the demands of society
The term “competencies” is used, as the magic word to express what is needed to
function effectively in a given social and cultural context. Clearly this question goes
beyond that of education responding to the demands of the labour market. Similarly
the concept of “competency” is meant to have a broader meaning than the
transmission of knowledge and skills in institutionalised education settings, like
schools. “Key competencies are considered as structured around meeting demands of
a high degree of complexity and are comprised of cognitive as well as motivational,
ethical, social and behavioural components” (Weinert, 2001, cited by Rychen, 2001,
p.11). It is recognized that acquiring these competencies also happens in other settings
than formal schooling, e.g. in the home situation of children and in the work-place
(learning on the job).
As the economic approach already illustrates competencies are identified by analyzing
valued “real life” situations, such as the functioning of highly schooled individuals in
work situations. Taking a broader outlook this valued situation could be described as
the “effective” or “competent functioning” of an individual in a given society, i.e. a
given cultural, social and linguistic context (Haste, 2001). This author identifies five
“competency domains” which she feels are essential in the immediate future and
possibly universally relevant. These are: “technological competency; the ability to
deal with ambiguity and diversity, the ability to find and sustain community links; the
management of motivation, emotion and desire; and finally, the sense of agency and
responsibility”. (Haste, 2001).
Apart from these “comprehensive” characteristics (cognitive, motivational, social and
ethical components), Rychen, ( 2001, p. 11,12) mentions two other core elements in
current, multi-disciplinary based, definitions of competencies:
- “Key competencies are seen as transversal or generic in the sense that they enable
individuals or groups to participate effectively in all relevant social fields with
their specific power and social relations, challenges and capital at stake.
29
-
Key competencies call for a higher order of mental complexity involving an active
and reflective approach, which includes the capacity to distance oneself from
one’s own socialising process and even one’s own values”.
When approached from the context of formal schooling, the idea of competencies can
be placed on a continuum of types of educational outcomes, already introduced in a
preceding section, that runs from specifically content oriented to “content free”
personality traits. Discrete positions on this continuum are presented in Figure 4.
-
outcomes as measured by tests included in textbooks
outcomes as measured by implemented school curricula (teacher developed)
outcomes as measured by tests based on the intended national curriculum
outcomes as measured by international tests covering the common core of a
range of national curricula, e.g. TIMSS
“literacy” tests, aimed at measuring basic skills in reading, mathematical and
scientific reasoning, e.g. PISA
competencies as multi-facetted dispositions of individuals, including cognitive,
motivational and possibly other components
personality traits, like internally or externally determined locus of control,
independence, general intelligence
Figure 4: A continuum of educational outcomes, running from highly content bound to
personality dependent
It remains to be seen whether all “ambitions” inherent in the idea of key-competencies
can be met. A core question is whether the multiple component idea of competencies
survives empirical measurement. For example in a current dissertation study an
instrument was developed to measure “career developmental competencies”,
involving cognitive, behavioural and motivational components. In analyzing the
instrument it appeared that these three components were relatively independent, and
thus could hardly be seen as forming one integrated construct.
Perhaps the heuristic value of the term will be that attention is drawn to other than
purely cognitive educational outcomes, whether these can be shown as an integral part
of a particular competency or as independent dimensions.
The practical meaning of the competency issue is that the answer to an effective
linking of education to societal functioning should partially be sought in the teaching
of more general skills and in paying more attention to motivational and attitudinal
aspects. Also the idea of meta-skills, such as general problem solving skills and
“learning to learn” could stimulate a more reflective and self-steering attitude of
30
future citizens. In this way “competencies” are considered as useful tools in serving
the “employability” of future citizens.
Overview of the most important categories of outcome indicators
In the table below, Table 3, an overview is given of the different categories of
outcome indicators that could be used to monitor quality aspects related to the
productivity, the effectiveness and equity of education.
Table 3: Overview of educational outcome indicators
Main categories of
outcome indicators
Sub-categories
Technical issues
Output indicators
Achievement measures
- subject matter based
- literacy (reading,
mathematical, scientific)
- competencies (e.g. learning
to learn)
- Value-added effect
measures; growth curves
- Assessment methodology
(ranging from multiple
choice tests to authentic
assessment)
- Criterion versus normreferenced testing
Outcome/attainment
indicators
Attainment measures
- graduation rates
- proportion of students
graduated without delay
- drop-out rates
- class repetition rates
Controlling for selection
oriented school policies
Impact indicators
Social participation rates
- (for each attainment level) %
of employed at a certain job
level)
- % of unemployed
- (for lower school levels: %
enrolled in follow-up
education
- degree of social participation
(social capital)
- adult literacy rates
- average income, for each
attainment level; earning
differentials
Availability of national
educational and labour
market statistics
31
Appropriate measures of
social capital and adult
literacy
-
skill shortages and surplus
Process indicators
Although indicators on educational processes can be use in a “stand alone” way,
according to what was earlier described as a disjointed application of indicators, it
makes more sense to see them as part of indicators systems, in combination with
outcome, input and context indicators. In this section process indicators will be
considered within the framework of macro-level, i.e. national level, indicator systems,
and, secondly, as part of multi-level indicator systems, where transformation
processes at school level are at the centre.
System level indicator systems
The OECD Education Indicators project (INES – see the Education at a Glance
publications) uses the following categorization, which is evident from the table of
contents in the Education at a Glance Publications (OECD, 1998).
The main categories are:
A) The demographic, social and economic context of education (e.g. Literacy skills
of the adult population)
B) Financial and human resources invested in education (e.g. Educational
expenditure per student)
C) Access to education, participation and progression (e.g. Overall participation in
formal education)
D) The transition from school to work (e.g. Youth unemployment and employment
by level of educational attainment)
E) The learning environment and the organization of schools (e.g. total intended
instruction time for pupils in lower secondary education)
F) Student achievement and the social and labour-market outcomes of education (e.g.
Mathematics achievement of students in 4th and 8th grades, and Earnings and
educational attainment)
These 6 categories can be classified in various ways. The context-input-processoutcome scheme, as used throughout this chapter, is the most likely way to do so.
Accordingly category A is in the context domain, category B refers to inputs,
categories C, D and E refer to different interpretations of the process dimension, and
category F to an output/outcome dimension. See Figure 5.
32
Context
demographic, social and
economic context of
education
Output/Outcomes
Input
Process
financial and human
resources invested in
education
access, participation, progression
transition school to work
learning environment and organization
of schools
achievement
labour-market
outcomes
Figure 5: Ordering of the OECD-INES education indicator set, according to a
context-input, process and outcome scheme
In Figure 5 arrows between the boxes have been omitted since, only in a very loose
sense, these categories are expected to be linked in a causal way. In fact each category
is used in a descriptive sense and interrelationships between indicators have hardly
been analyzed so far.
A second way to look upon the OECD indicator set is by distinguishing “stock” and
“flow” types of indicators. Categories A and B are typically stock indicators, whereas
categories C and D, and to some extent F refers to flows.
A stock indicator describes a relevant educational aspect at one point in time in
quantitative and qualitative terms (e.g. the number of qualified teachers in school year
x). A flow indicator refers to the transition of an educational unit, e.g. a student or a
teacher, to a different part of the system (e.g. the number of teachers that have left the
profession in school year y).
It should be noted that classifying “transition” or “flow” indicators as process
indicators adds a third interpretation to process indicators. There are really three types
of process indicators around:
a) malleable conditions of transformation processes (see the systems model in
Figure 1);
b) checks on program implementation within the context of program evaluation;
c) flows of units through the educational system (OECD).
Since our subject is the quality of education, the first interpretation of process
indicators will be predominant. Indicators on access and participation are more related
to issues of the quantity of supply and demand of education. Success-rates and
survival rates, can be seen as belonging to the domain of participation indicators, but,
as was evident from the previous section, these kind of indicators are treated as
outcome measures of educational attainment. Similarly transition from school to work
is considered as an issue of the societal impact of education.
33
In Figure 5, where the OECD indicator set is categorized, process indicators seen as
malleable conditions may refer to two different types of information; to policy
measures and structures that are defined at the national level on the one hand, and to
aggregated school characteristics on the other.
In Table 4 below, some examples of process indicators that are actually defined at the
national system level are provided.
Table 4: Examples of system level process indicators
Process indicators defined at the level of national education systems













Teaching time per subject
Total hours of instruction per year, for specific grade levels in primary and secondary
education
Opportunity to learn, in terms of expert ratings of test curriculum overlap
The
locus
of
decision-making
in
education,
by
education
level
(This indicator shows at which administrative level decisions in sub-domains of
education – curriculum, personnel management – instruction, resources – are made with
a certain degree of autonomy)
School autonomy (this indicator is actually included in the concept of locus of decision
making)
Education standards by level [e.g. targets like increased completion rates, percentage of
students scoring at or above a particular achievement level].
Whether or not formal examinations are taken at the end of each school category
The degree of categorization and formal streaming at secondary level
The evaluation capacity of the system (defined as a quantification of the occurrence and
intensity of various evaluation forms, such as national assessment programs,
examinations, school inspection, an educational management information system etc.)
The magnitude and diversification of an educational support structure in the country
(possibly comprising a curriculum development unit, ICT services, school counseling, an
educational assessment and testing unit etc.)
The division of private, government dependent and public schools
Incentive based policies to stimulate school performance
The degree to which school choice is free.
It should be noted that system level process indicators may be interpreted as context
indicators when one takes the perspective of the school.
34
comprehensive indicator systems including process indicators of school functioning
aggregation levels
Educational systems have a hierarchical structure where administrative levels are
“nested”. Indicator systems may ignore this hierarchical structure by using statistics
that are defined at national level or formal characteristics of the system. Examples are:
pupil teacher ratio computed as the ratio of all pupils and all teachers in a country and
teacher salaries defined on the basis of nationally determined salary-scales.
Even when considering use of indicators at national level only, there are two main
advantages to use data at lower aggregation levels:
 disaggregated data allow for examining variation between units, e.g. the variance
between schools in success rates on examinations;
 disaggregated data allow for better adjustments and more valid causal inferences;
the best example in education is the use of so called “value-added” performance
indicators based on achievement test scores adjusted for prior achievement and/or
other relevant pupil background characteristics.
When it is the intention to relate, for example, school organizational characteristics to
pupil achievement, disaggregate data at pupil level are required to carry out
appropriate multi-level analyses.
Particularly when indicators are used for program evaluation purposes the above
mentioned advantages of disaggregate data are important, because they provide firmer
ground to answer causal questions about program effectiveness.
A final added advantage is that the relevance of indicator systems for lower
administrative levels (e.g. school districts and individual schools) grows when
disaggregate data is available.
Time-frame
For the purposes of evaluating large-scale educational development programs
experimental and quasi-experimental designs have been recommended. (Ezemenari,
Rudqvist & Subbarao, 1998).
Although there is no question about it that (quasi-)experimental designs should be
used whenever possible (compare Campbell’s famous idea of “Reforms as
Experiments, Campbell, 1969), they are often not feasible.
Using educational indicators in a longitudinal way, whereby the same units are
measured at several points in time, is a viable alternative to experimentation.
Function of process indicators
In previous sections various interpretations of educational process indicators were
referred to. In this section process indicators that reflect malleable conditions of basic
transformation processes at school level will be placed central (see Figure 1). School
organizational functioning and teaching and learning at classroom level are examples
of such educational transformation processes.
35
In general it could be said that such process indicators shed some light on what
happens in the “black box” of schooling. Process indicators are interesting from the
point of view of policy and management since they refer to conditions that are
malleable and thus the subject of active policies to improve education.
In a later chapter the perspective of school effectiveness research will be presented as
the most likely rationale for identifying and selecting process indicators. Accordingly
those process indicators are selected that show positive associations with educational
output and outcomes.
Ideally such process indicators should be able to predict output (as in “education
production functions”: increments in “process” conditions predicting increments in
output according to an exact function). To the extent that such instrumental
knowledge would be complete process indicators could rightly be used as substitutes
of output indicators. Given the fact that the education production function is debated
and, more generally, school effectiveness knowledge is “incomplete”, to say the least,
such a strong instrumental interpretation is not realistic.
This leaves two further possibilities for the use and interpretations of process
indicators:
 as “annex” to output indicators, whereby in each and every situation of their use
the association between process and output indicators would have to be explored
with the intention to “explain” differences in outcomes between schools and
between educational systems;
 a weaker interpretation of instrumentality, where process indicators are seen as
instance of educational good practice, and, in this way, could lead to valuejudgements about educational quality even in the absence of output data.
Within the context of program evaluation process indicators are sometimes defined as
checks on the actual implementation of the program. This interpretation of process
indicators is easily reconcilable with the one used throughout this section.
Implementation checks are a more basic and administrative type of monitoring,
whereas process indicators as defined above, are referring to more generic causal
processes of organizational functioning and teaching and learning. When process
indicators are used over and above implementation checks, they say more about why
an (implemented) program works. Figure 6 illustrates this.
program
inputs
degree of
implementation
of program
inputs
transformation
processes
following program
implementation
Figure 6: Use of process indicators in the context of program evaluation
36
outputs
School effectiveness research and the identification of process indicators
The most likely rationale for selecting process indicators is to choose those variables
that are manipulative predictors of school output. Research literature on school
effectiveness can be used as a source to identify promising process variables.
Generally speaking, school effectiveness research is aimed at discovering school
characteristics that are positively associated with school output, usually measured as
students' achievement. Various research traditions can be subsumed under this
heading, including (in)equality of education (sociological), educational production
functions (economical), school improvement and effective schools, and teacher- and
instructional effectiveness (psychological). Apart from these, more theoretical and
analytic contributions from organizational science and micro-economic theory of
public-sector organizations can also be sources of inspiration in selecting process
indicators (see Cameron & Whetten, 1983 and Niskanen, 1971, respectively).
For reviews of the school effectiveness research literature see Scheerens & Bosker,
1997 and Scheerens, 2000. More details about the research approach and substantive
results of school and instructional effectiveness research will be provided in Chapter
3.
In the overview of examples of process indicators on school functioning this research
literature has been the predominant source. Table 5 summarizes effectiveness
enhancing factors at school level, and Table 6 shows variables at the level of teaching
and learning which have been associated with effectiveness.
37
Table 5: Overview of examples of process indicators of school functioning
Process indicators defined at school level
Community involvement
 the degree of actual involvement of parents in various school activities (the teaching and
learning process, extra-curricular activities and supporting activities)
 the percentage of the total annual school budget that is obtained from the local
community
 the amount of discretion local school boards have in the conditions of labour of teachers
[possible operationalizations in EDUCO project – El Salvador]
School financial and human resources
 average years of teachers’ experience per school
 school level pupil teacher ratio
 average class size per school
 proportion of formally qualified teachers per school
 school managerial “overhead” (principal and deputy-principal fte per 1000 students)
Achievement oriented policy
 whether or not schools set achievement standards
 the degree to which schools follow (education) careers of pupils after they have left the
school
 whether or not schools report achievement/attainment outcomes to local constituencies
Educational leadership
 the amount of time principals spend on educational matters, as compared to
administrative and other tasks
 whether or not principal’s appraise the performance of teachers
 the amount of time dedicated to instructional issues during staff meetings
Continuity and consensus among teachers
 the amount of changes in staff over a certain period
 the presence or absence of school subject-related working groups or departments
(secondary schools)
 frequency and duration of formal and informal staff meetings
Orderly and safe climate
 statistics on absenteeism and delinquency
 ratings of school discipline by principals, teachers and pupils
Efficient use of time
 total instruction time and time per subject matter area
 average loss of time per teaching hour (due to organization, moving to different rooms,
locations, disturbances)
 percentage of lessons “not given”, on an annual basis
Opportunity to learn
 teacher or student ratings of whether each item of an achievement test was taught or not
38
Evaluation of pupils’ progress
 the frequency of use of curriculum specific tests at each grade level
 the frequency of use of standardized achievement tests
 the actual use teachers make of test results
Ratings of teaching quality
 quality of instruction as rated by peers (other teacher)
 quality of instruction as rated by students
Table 6: Overview of effective teaching and learning variables
Effective teaching variables
Main teaching factors
 opportunity to learn
 structuring and scaffolding (cognitive structuring)
 stimulating engagement (motivational structuring)
 climate aspects:
- task orientation
- mutual respect
- orderliness, safety
 monitoring and questioning
 feedback and reinforcement
 modeling learning and self-regulation strategies
 “authentic” applications
 adaptive teaching
Learning strategies of students
overt: engaged learning time
student use of resources
cooperative learning
covert: self-regulatory capacity
auto-control
meta-cognitive “actions”
learning styles
39
Input indicators
According to our basic input-process-outcome-context framework inputs provide the
material and immaterial pre-conditions for the core transformation processes in
organizations. In the case of education, and taking the school as the level where
teaching and learning as the primary transformation process take place, the following
main categories of inputs can be discerned:
- financial and material resources
- human resources
- background conditions of the students
Household and community characteristics as well as guidelines and operational
frameworks from administrative levels above the school might also be considered as
inputs, when the school is chosen as the level of analysis. The latter have been
described in this chapter under the heading of system level processes and system level
process indicators. The former (household and community characteristics) appear to
be better classified as “context” characteristics, particularly when the environment
provides given antecedents and constraints rather than active control measures. These
choices, however, are quite arbitrary. For out purposes the framework primarily
serves a heuristic function aimed at providing a comprehensive overview of indicators
that can be used to measure a broad range of aspects of educational quality.
Input indicators contribute to assessing the quality aspects of effectiveness (which
inputs “work”), efficiency (which inputs work at the lowest possible price) and equity
(equitable distribution of material and human resources inputs to schools, the degree
to which outcome levels depend on student background characteristics).
Financial and material resources
Financial and material resources indicators can be defined at system and at school
level. Financial indicators are predominant at system level, while material resources
indicators make more sense to be formulated at the level of schools. In Table 7 below
examples of financial and material resources indicators, defined at system and at
school level are given.
40
Table 7:
System level financial and material resources indicators










proportion of Gross Domestic Product spent on education
educational expenditure per student
proportion of public and private investments in education
public investment in educational research and development
total expenditure on programs and special facilities for disadvantaged students
state provision of ancillary services
household expenditure and public subsidies to parents
percentage of spending on salaries for administrative personnel
percentage of spending on pensions for educational personnel
percentage of spending on salaries for teachers
School level financial and material resources






proportion of the school’s budget that is acquired through other than public funding
school building facilities
classroom equipment (furniture, computers, etc)
school supplies like pencil and paper, chalk board, flipchart
availability of textbooks in the major school subjects
basic services like separate toilets for girls and boys, water, electricity, heating,
telephone, provision of ancillary services, regarding nutrition, health and transportation
Human resources
A well- qualified and motivated teaching force is to be seen as one of the most vital
assets for educational quality. Indicators on teachers as individuals or of the total
stock of teachers in a country can be categorized in various ways.
In the table below a distinction is made between descriptive background
characteristics of teachers, knowledge and skills, attitudes and morale relative to
general working conditions and attitudes with respect to the work situation at school
and student staff ratios.
41
Table 8: Categorization of teacher indicators.
Teacher background characteristics








Age, sex, and ethnicity distribution
full-time/part-time distribution
certification/license status
formal qualifications
year of experience
language
health, specifically HIV
in-service training history
Teacher professional knowledge and skills






general knowledge
content knowledge
knowledge about pedagogical and didactic strategies
knowledge about students
beliefs and attitudes about teaching
flexibility in adapting teaching repertoire
Teacher working conditions











salaries (relative to other professionals)
working time
average class-size
merit based incentives
other incentive policies
career structures
teacher training/certification requirements
teacher autonomy
standards-based teacher appraisal
secondary working conditions (e.g. vacations)
exposure to external inspection
Teacher morale and status








opinions about career and job mobility
teacher morale
perception about being needed by society
perceived status as a teacher
appreciation of general working conditions
appreciation of the work situation at school of current employment
job mobility
sense of political efficacy
Staff to student ratios
42




system level student teacher ratio
school level student teacher ratio
support staff student ratio (system and school level)
school managerial “overhead” relative to the number of students
Only a part of these indicators is likely to be available on the basis of national
statistics, and would depend on the availability of school or teacher surveys.
Student background characteristics
To some perceiving students as the “raw material of the education production
process” may seem to stretch the economic metaphor a bit far. More psychologically
inclined analysts might maintain that students are the main producers of learning and
the attainment of learning results. For analytical purposes it makes nevertheless sense
to recognize that the home background and intellectual capacities of students make a
lot of difference. When effectiveness and productivity interpretations of quality are at
stake it is usually considered relevant to construct value-added outcome indicators,
that is indicators that show the effect of malleable conditions of schooling over and
above the impact of background conditions. For equity interpretations of quality
student background characteristics function as categorization criteria, to contrast
groups with one another, for example, boys and girls, schools with a relatively small
and a large proportion of students from minority groups, etc.
As will be documented more fully in chapter 4, when presenting results of empirical
school effectiveness research, student background variables do not only have an
impact as individual student characteristics but, when considering the average
achievement level of a class or school, also as a so-called compositional effect. This
means that average achievement is influenced by the composition of the group in
terms of intellectual capacity and socio-economic background.
For our purposes a distinction is made in two categories of student background
characteristics. The first is selected because of its more general association with
educational performance, the second refers to specific circumstances, particularly in
developing countries, that are likely to interfere with a student’s capacity to optimally
benefit from schooling.
43
Table 9:
General student background characteristics
-
general intelligence or scholastic aptitude
socio-economic status
mother’s level of educational attainment
gender
ethnicity
Student background characteristics associated with specific situational constraints
-
discrepancy between language spoken at home and language at school
distance a student has to walk to school
the amount of out of school time a student has to spent on labor
whether the students has had a meal when arriving at school
place to study at home
number of books in the home
malnutrition
ill health/HIV
Contextual issues and context indicators
What has already been covered
Within the input-process-outcome-context framework specifying what is meant by the
context depends on the level at which the central transformation process is defined.
Throughout this chapter two interpretations have been used. Most of the time
transformation processes at school level have been concentrated on. When
transformation processes at school are further differentiated to distinguish primary
teaching processes at classroom level and secondary, supporting management and
organization processes at school level, a multi-level model results, in which
everything “outside” the school is defined as the context. In this kind of
conceptualization “context” could be further subdivided in the direct environment,
local community and local/regional administration on the one hand, and the national
context on the other. The second model interpretation that has been referred to now
and then in this chapter is the one where education is considered at one level only, the
national system level. According to this interpretation the context is defined as the
relevant environment of the “education province” as a whole. As such the general
affluence of a country, demographic tendencies, cultural aspects that impinge on
values that are important in education and the institutional infrastructure of a nation
could be seen as the context of education.
44
So far “malleable” conditions at the national level have been described already, and
from the perspective of the school these form an important category of contextual
conditions. Particularly the patterns of functional (de) centralization, which result
when one determines the locus of decision-making for particular decisional domains,
and the relative autonomy with which decisions are taken at a certain level, is quite
fundamental for the functioning of schools. It includes for example the question of
centralization and decentralization in the curriculum domain. Other important
nationally malleable conditions that were mentioned were: the degree of development
of evaluation capacity throughout the system, and the question of explicit standards
and formal examinations.
Having covered this important category of school context indicators, what are left are
maybe a few antecedent conditions at national level, and a few antecedent conditions
at the level of the school community. As stated before the, sometimes relative,
distinction between “malleable conditions” and “given, antecedent conditions”
expresses the fact that some conditions have a more permanent status and if at all
changeable, this would be more in the sense of gradual evolution than by means of
active steering operations. So, a closer look will be taken at these remaining areas of
context indicators. Firstly, given conditions in the society at large that are relevant for
the educational system as a whole, secondly, given conditions that are part of the
national education system, and, thirdly, given conditions in the immediate
environment of the school.
When society at large is considered as the context of the “education province”
Obviously when dealing with the whole of society it is totally beyond the scope of
this presentation to do a comprehensive analysis. Instead some areas and dimensions
will be mentioned that are believed to imply relevant contextual constraints for the
education system.
Demographic information concerning the magnitude of age cohorts and longer term
projections of these indicate the scope of education in terms of the number of students
that need to be serviced over time.
Labor market statistics concerning employment rates, wage differentials and skill
shortages and surpluses in different sections of the labor market. It should be noted
that these kind of indicators can also be analyzed as impact indicators. The purpose
and time dimension makes the difference. When such indicators are used pro-actively
in the context of planning educational provisions they can be seen as part of the
relevant societal context. In the case of evaluating the impact of the functioning of the
education systems these indicators are used retrospectively and therefore can also be
classified as a particular kind of outcome indicator.
General information on the economy, such as indicators on the affluence/poverty of a
country, is obviously relevant in indicating the likely margins of educational
expenditure.
45
Aspects of the culture and cultural tradition of a country. The value that education has
in a particular cultural tradition will vary as well as norms and values that bear upon
the expected role behavior in education, e.g. the way authority of teachers and the
participation of students are seen. Another example could be the degree of trust in
others. As the World Value Study (1990) has shown, there are enormous differences
in the percentage of people in a country that answer yes to the question whether or not
most people can be trusted (in Norway this is 65%, in South Africa 18% and in
Turkey 6.5%; source, OECD, 2001). Perhaps it would also be interesting to make
more systematic study of the implications for education of Hofstede’s categorizations
of national cultures Hofstede, (1980). Hallinger, (2001) provides a fascinating
analysis of the way educational leadership and school improvement are interpreted in
the culture of Thailand, which he characterizes in terms of Hofstede’s dimensions of
“power-distance”
“masculinity–femininity”,
“uncertainty
avoidance”
and
“collectivism”. Cultural values that express support and reverence for education and
“translate” into a high appreciation of education and a high status for teachers are
often seen as explanatory for the success in education of countries like Japan and
Finland.
The institutional infrastructure. New institutional economics stresses the importance
of institutions. Institutions not just in the sense of organizational provisions, but also
seen as “the rules of the game”, North (1990) defines institutions as the broad
complexes of rules and norms that govern the behaviour of individuals, organizations
and firms. They include legal, political and regulatory systems, the policing
mechanisms for enforcing laws, financial and insurance systems, a free press, the
market system and all the rules and formal and informal enforcement mechanisms that
operate within it. (McMeeking, 2003, 11). The degree to which education in a country
has become institutionalized is likely to depend on the institutionalization of the
society at large. And the rights and liberties of teachers, for example, are likely to
conform to those of personnel in the public service at large. Alternately the eventual
lack of institutionalization in the society at large could also be reflected in the
education system, and this could bear upon important disciplinary issues like the
degree to which teacher absenteeism is tolerated.
The general health situation in a country. When relative large parts of the population
suffer from illnesses, like HIV/aids, this will clearly have a detrimental effect on the
functioning of education as well.
Disasters of nature and war. Finally there is the impact of disasters of nature, for
example floods that happen more frequently as the probable consequence of global
warming, and of countries being in a state of war.
The table below summarises these types of societal conditions.
Table 10 Types of societal conditions particularly relevant to education
46
Contextual conditions of education systems
. Demographic developments
. The labour market, e.g. shortages and surplus in certain sectors
. The general state of the economy
. Relevant cultural aspects
. The institutional infrastructure
. The general health situation in a country
. Disasters of nature and war
Antecedent conditions at national level within the education sector
Some of the societal dimensions mentioned in the section above have specific
translations to conditions within the educational system. An overview of areas
relevant for description and indicator development is given in the table below.
47
Table 11: Antecedent conditions within the educational system
Demographics




the supply and demand of teachers in a country
the proportion of teachers over 50 years old
the gender composition of the teacher force per school level
percentage of students in school outside the age ranges for grade levels
Cultural aspects




the status of teachers as perceived by the general public
appreciation of education and being educated
expectations about pedagogical functions of the school (e.g. educating for good
citizenship, moral education, teaching democracy)
cultural embedded interpretations relative to authority and educational leadership
Institutional infrastructure







degree of formalization of teacher working conditions
formalization of teacher, student and parent rights (e.g. free school choice)
formal monitoring and inspection of schools
rules and enforcement of rules with respect to teacher absenteeism
regulations with respect to private tuition by teachers in the public service
anti corruption measures in education
framework for delivering and assessing the curriculum
Context indicators at the level of the local community
In earlier sections the more active involvement of the community has already been
referred to, by mentioning community involvement and parental involvement with a
particular school. Again there is a vague line that separate theses “malleable
conditions” from other conditions that are considered as given. As major categories to
classify contextual conditions in the local school setting organizational infrastructure
and (local) culture will be used.
48
With respect to the organizational infrastructure the factors summarized in the table
below (Table 12) appear to be relevant:
Table 12: The organisational infrastructure of the local community
The organisational infrastructure of the local community
-
the existence of a school board in which the local community is represented
the availability of a local or regional educational resources centre (which, among others,
might offer ICT facilities to the schools in the community)
the “openness” of local companies and industry to work with schools and offer students
opportunities for site visits or specific training opportunities
the role of the community in financing the school; in-kind support
Cultural aspects that are manifest in the local community, are likely to reflect
regional, national or even “world cultural” traditions. It is therefore somewhat
arbitrary to deal with these conditions at national or local level. The reason to do so
here, is that the local level is the one closest to the school, and the aspects to be dealt
with are seen primarily as contextual constraints on the functioning of schools. Fuller
and Clarke, (1994) have distinguished different types of cultural constraints relative to
the effective functioning of schools.
The first category of “local cultural conditions” distinguished by Fuller and Clarke
concerns “prior effects of family processes an current effects from labor demands
placed upon children”. They mention a range of variables that appear to be quite
plausible for their relevance in developing countries. Examples are: ethnic
membership, perceived legitimacy of the school by the parents, commitment of
parents to literacy, willingness of parents to engage in modern child health and
nutritional practices, gender, and distance to school.
The second type of local condition treated by Fuller and Clarke pertains to the
correspondence or rather the distinction between indigenous knowledge and “school
knowledge”. They refer to the common finding that knowledge that is to a greater
degree exclusively dealt with at schools usually shows larger effect of schools and
school conditions as compared to knowledge that is also acquired at home
As a third category, Fuller and Clarke refer to discrepancies between “participatory
forms of pedagogy” which “Western policy mechanics” “would like” to show as
being effective regardless of cultural setting. The problem they discern is that
participatory forms of teaching may be at odds with hierarchical forms of teacher
authority that coincide with cultural traditions.
49
The “culturally constructed meaning of classroom inputs”, which is the fourth type of
local condition discussed by Fuller and Clarke, relates to the phenomenon that inputs
of schooling, such as textbooks are interpreted and adapted in the ways they are
actually used by teachers. This phenomenon is also well known in “Western”
curriculum studies, where the distinction between an “intended” an “implemented”
curriculum has similar implications. The additional element in developing countries
being the interaction of traditional culturally embedded ways of teaching with the
classroom inputs concerned. It makes sense to expect that these “mediations” of
school inputs will be stronger to the degree that the intended use differs from the
established teaching traditions. Fuller and Clarke provide examples from developing
and industrialized countries of the pervasiveness of preferred ways of teaching e.g.
expository teaching, despite the explicit hints and guidelines in textbooks to practice
discovery learning. An example from education production function research in
industrialized countries that also illustrate the different meanings and implications of
school inputs in different cultures is the difference in the way class size is dealt with
by teachers in Japan as compared to the US (Galton, 1999).
In our quality framework the meaning of these cultural conditions, manifest in the
local school community, is that they are likely to interfere with active measures to
enhance school effectiveness, which might be inspired by an international common
understanding of “what works” in education. The four types of cultural constraint,
distinguished by Fuller and Clarke are summarized in Table 13, below.
Table 13: Overview of local cultural conditions as examples of given local contextual
conditions.
Local cultural conditions
-
parents’ values concerning school participation of their children
-
discrepancy between indigenous knowledge and “school knowledge”
-
discrepancy between local perspectives on authority and ideas on active participation of
students during lessons
-
culturally constructed meaning of school inputs
Mechanisms of responsiveness at system and at school level
In our quality framework “context” has been interpreted as “provider of direct
influence and control”, as a “provider of inputs” and as a source of more general
“constraints” that interfere and interact with more direct control measures. In all of
50
these interpretations the direction of influence is from the context to the educational
level, be it the school or the national educational system, that is being described or
analyzed for its quality. However, in one of the quality perspectives that were
described in the chapter, the direction is the other way around. According to the
responsiveness perspective schools or national educational systems attempt to meet
the demands of the relevant context, the local community or society at large. This
means generally two things, first, that the intended outcomes or the goals and
objectives are in line with the external expectations and second, that these goals are
also actually being realized. The latter issue is the effectiveness issue, while the
former is properly addressing the question of the responsiveness of educational
organizations. The key question in the responsiveness interpretation of educational
quality is therefore whether the “right” goals are chosen as a first step to delivering
what is externally required. This is not a simple matter. In the most straightforward
application, the issue of vocational education meeting the demands of the labour
market, the complexity of the issue becomes painfully clear. Methods that seek to
make explicit the skills and knowledge for a particular field of application and use
such job profiles as a basis for establishing curriculum goals are indicated as “naïve”.
For our purposes the main question is whether an educational system has an
infrastructure and established mechanisms to deal with responsiveness questions. Not
only with respect to the demands of the labour market, but also with respect to other
kinds of societal demands to education, as for example developing good citizenship.
At national level this focuses the attention on the availability of an institutional
infrastructure for curriculum development, including procedures for controlling the
actual implementation of nationally agreed curriculum priorities. In order to fulfill the
responsiveness function well, such a curriculum development infrastructure should
have established liaison functions with industry and other societal organizations.
Sometimes the design of the school structure can include conditions that facilitate
responsiveness, as in the case of so-called dual vocational education (forms of
education where the students spent part of their school time in companies).
At school level, an active approach of school management to maintain contacts with
relevant local organizations, as well as with parents, could be seen in line with the
responsiveness criterion.
Areas for describing “responsiveness to context” at system and school level are
summarized in Table 14 below.
51
Table 14: Areas of responsiveness to context at system and school level
Areas for describing responsiveness to context at system level
-
the availability of an institutional infrastructure for curriculum development
enforcement mechanisms that monitor curriculum development and implementation
liaison functions of educational authorities and societal organizations
analysis and research units that try to predict the demands of the labor market
dual systems in vocational education
Areas for describing responsiveness of the school towards the local community
-
external contacts of school management
“school marketing policies”
active role of the school in acquiring parental involvement
“authentic” teaching examples involving representatives from the local community
Summary and conclusion
In this chapter the well-known input-process-outcome-context framework was used to
define different perspectives on educational quality: productivity, effectiveness,
efficiency, equity, responsiveness and a more eclectic use of quality indicators. The
framework was critically analyzed with respect to its dynamic application of planning
and controlling education. The flexibility and broadness of the framework was
documented by describing modifications of the “rationality model” that increasingly
address the complexity, situational contingency and interactive nature of social
reality. The fact that the basic categories of the framework stand up as being useful
descriptive categories for these modified interpretations as well was taken as a further
encouragement to the use of the framework as a categorization scheme for indicators.
Input, process, outcome and context education indicators that were further described
and specified in the remaining part of the chapter. The indicator set has been
summarized in the Table 14 below.
52
Table 15: Synthetic overview of educational input, process, outcome and context
indicators
INPUTS
System level financial and
material (see Table 7) and
human resources (see
Table 8) indicators
PROCESS
System level process
indicators
See Table 4
School level financial and
material (see Table 7) and
human resources (see
Table 8) indicators
School level process
indicators
See Table 5
Student background
characteristics
(See Table 9)
Effective teaching
variables
See Table 6
OUTCOMES
Output indicators
 subject matter based
 literacy (reading,
mathematical, scientific)
 competencies (e.g. learning
to learn




Areas for describing
responsiveness
See Table 14
Contextual factors
Antecedent conditions
at national level
 demographics
 cultural aspects
The societal context
 demographic information
 aspects of the culture and cultural
tradition of a country
 economic aspects
institutional infrastructure
 the institutional infrastructure
(see Table 11)
 the general health situation
in a country
 disasters of nature and war (Table 10)
Outcome/attainment
indicators
graduation rates
proportion of students
graduated without delay
drop-out rates
class repetition rates
Impact indicators
 (for each attainment level) %
of employed at a certain job
level
 % of unemployed
 (for lower school levels) %
enrolled in follow-up
education
 degree of social participation
(social capital)
 adult literacy rates
 average income, for each
attainment level
Context indicators of
the local community
 the organizational infrastructure
 local cultural conditions
(see Tables 12 and 13)
References
Argyris, C., & Schön, D.A. (1974). Theory in practice: increasing professional
effectiveness. San Francisco: Jossey-Bass.
53
Bosker, R.J., & Velden, R.K.W. van der (1989). Schooleffecten en rendementen. In:
J. Dronkers & J. van Damme (Eds.), Loopbanen doorheen het onderwijs. Lisse:
Swets & Zeitinger.
Bosker, R.J., & Witziers, B. (1996). The magnitude of school effects. Or: Does it
really matter which school a student attend. Paper presented at AERA Annual
meeting, New York.
Brandsma & Doolaard (1996)
Cameron, K.S., Whetten, D.A. (eds.) (1983). Organizational Effectiveness. A
comparison of multiple models. New York: Academic Press.
Campbell, D.T. (1969). Reforms as Experiments. American Psychologist, 24(4).
Cheng, Y.C. (1993). Conceptualization and measurement of school effectiveness: An
organizational perspective. Paper presented at AERA annual meeting, Atlanta,
GA.
Cohen, M.D., March, J.G. & Olsen, J.P. (1972). A garbage can model of
organizational choice. Administrative Science Quarterly, 17, 1-25
Ezemenari, K., Rudqvist, A., & Subbarao, K. (1998). Impact evaluation: a note on
concepts and methods. PRMPO World Bank, Washington D.C.
Fuller, B., & Clarke, P. (1994). Raising school effects while ignoring culture? Local
conditions and the influence of classroom tools, rules and pedagogy. Review of
Educational Research, 64, 119-157.
Galton, M. (1999). Class Size and Pupil Achievement. International Journal of
Educational Research, vol. 29.
Goldstein, H. (1996). Multilevel statistical models. London: Arnold.
Goldstein, H., & Thomas, S. (1996). Using examination results as indicators of school
and college performance. Journal of the Royal Statistical Society. Series A ,
Statistics in Society, vol. 159, no. 1, 149-164.
Hallinger 2001
Haste, H. (2001) Ambiguity, Autonomy, and Agency: Psychological Challenges to
New Competence. In: Rychen. D.S., & Hersch Salganik, L. Defining and
selecting Key Competencies. Seattle, Toronto, Bern Goettingen: Hgrefe & Huber
Publishers.
Hausman, J.A., & Wise, D.A. (1979). Attrition bias in experimental and panel data:
the Gary Income Maintenance Experiment. Econometrica, 47, 455-473.
Hirsch, D. (1994). School: A matter of choice. Paris: OECD/CERI.
Hofstede ,P., (1980) Culture’s consequences: International differences in workrelated values. Beverly Hills, CA:Sage
Hoogerwerf, A. (1978) Overheidsbeleid (Public Policy). Alphen aan den Rijn:
Samson
Leune, J.M.G. (1994). Onderwijskwaliteit en de autonomie van scholen. In B.P.M.
Creemers (red). Deregulering en de kwaliteit van het onderwijs. (pp. 27-48)
Groningen: RION.
54
Levy, F., & Murnane, R.J. (2001) Key Competencies Critical to Economic Success.
In: Rychen. D.S., & Hersch Salganik, L. Defining and selecting Key
Competencies. Seattle, Toronto, Bern Goettingen: Hgrefe & Huber Publishers.
Lockheed, M.E. (1988). The measurement of educational efficiency and effectiveness.
AERA paper. New Orleans.
Luhmann, N. (1995). Social systems. Stanford, Ca.: Stanford University Press.
Madaus, G.F., Kellaghan, Th., Rakow, E.A., & King, D.J. (1979). The sensitivity of
measures of school effectiveness. Harvard Educational Review, 49, 207-230.
March, J.G. & Simon. H. A. (1958) Organizations Jon Wiley: New York
McMeeking, R. M., (2003) Incentives to improve education. Cheltenham UK,
Northampton MA, USA: Edward Elgar
Meuret, D. & J. Scheerens (1995). An International Comparison of Functional and
Territorial Decentralization of Public Educational Systems. Paper presented at
AERA 1995, San Francisco.
Morgan, G. (1986). Images of organizations. Beverly Hills: Sage.
Niskanen, W.A. (1971). Bureaucracy and representative government. Chicago: AldineAtherton.
North, D.C., (1990) Institutions, Institutional Change and Economic Performance.
Cambridge UK: Cambridge University Press
OECD (1998). Education at a Glance. Paris: OECD.
OECD (2001). Knowledge and Skills for Life. First results from PISA 2000. Paris:
OECD.
Purkey, S.C., & Smith, M.S. (1983). Effective schools: a review. The Elementary
School Journal, 83(4), 427-452.
Raudenbush, S.W., & Willms, J.D. (1996). The estimation of school effects. Journal
of Educational and Behavioural Statistics, 20, 307-335.
Reneman, D., (2000) Self-Reference and Policy Success Dissertation: University of
Utrecht, the Netherlands
Riley, D.D. (1990). Should market forces control educational decision making?
American Political Science Review, 84, 554-558.
Rutter, M., Maughan, B., Mortimore, P., & Ouston, J. (1979). Fifteen thousand hours.
London: Open Books.
Rychen, D.S. (2001) Introduction. In: Rychen. D.S., and Hersch Salganik, L. Defining
and selecting Key Competencies. Seattle, Toronto, Bern Goettingen: Hgrefe &
Huber Publishers.
Scheerens, J. (1990). School effectiveness research and the development of process
indicators of school functioning. School Effectiveness and School Improvement,
1(1), 61-80. Lisse: Swets & Zeitlinger.
Scheerens, J. (1992). Effective Schooling, Research, Theory and Practice. London:
Cassell.
Scheerens, J. (2000). Improving school effectiveness. Paris: UNESCO, IIEP,
Fundamentals of Educational Planning series no. 68.
55
Scheerens, J., & Bosker, R.J. (1997). The Foundations of Educational Effectiveness.
Oxford: Elsevier Science Ltd.
Scheerens et al 2002 bequad?
Simon, H. A. (1945) Administrative Behavior. New York: Mc. Millan
Weinert, F.E. (2001). Concept of competence: a conceptual clarification.. In: D.S.
Rychen, & L.H. Salganik (Eds.), Defining and selecting competencies, 45-64.
Seattle: Hogrefe & Huber Publishers.
56