Diversity beyond context?
Methodological tensions in applying ethnography in evaluation
Stijn Suijs
KU Leuven, Department of Educational Sciences
Abstract
In 1995 the Centre for Intercultural Education was founded at the University Of Ghent (Dutch-speaking part of
Belgium). The very first research project, intensive school ethnography, deeply influenced the concept of
intercultural education of the newborn Centre.
This concept very much relied on assumptions of the ethnographic tradition, especially the inclination to an
ongoing social construction of ‘reality’. From this perspective intercultural education can be different in each
specific situation as the implied meaning of diversity can shift from context to context. I joined the Centre at the
moment the research team engaged in evaluation research requiring overall context generalizations. The
challenge then was to maintain a context-related dynamic perspective in a research design more familiar with the
empirical-analytical tradition than with naturalistic research. We made important concessions to the demands of
the commissioners of the studies, undermining essential assumptions from the ethnographic tradition. These
assumptions are elaborated in the first section of the article. The next chapters explore the tensions in applying
these assumptions in evaluation research. Evaluation supposes a clear definition, if not a theory about the subject
under investigation. On top the commissioners demanded general statements, going far beyond the observation
of the particular individual teacher in his specific context. I claim that the problem of theory (section 2) and
generalization (section 4) are not proper to the confrontation of inquiry cultures in our research, but are already
inherent to tensions within the ethnographic tradition. Section 3 presents a comparison of two of the involved
evaluation studies. The first of them is presented as an example of the incompability of competing inquiry
paradigms. The second one on the other hand shows an interesting attempt to re-introduce context-awareness in a
context-hostile research design. Each of the discussed evaluation studies applied ethnographic techniques and
concepts. But the application of techniques and concept does not suffice to regard the basic assumptions of the
ethnographic tradition. Ethnographic inspired research should at least take the different and shifting
interpretations of the evaluated program into consideration. But if in the discussed studies the different
interpretations of participants were taken into account, than only to measure to which account they met a
‘correct’ understanding of the program. As such the evaluation studies neglect the dynamic of shifting meaning
in a multi layered cultural reality. And they even negate the own definition of intercultural education as an open
ended process.
Résumé
L'année 1995 a vu la création du Point d'Appui de l'Education Interculturelle à l'Université de Gand
(Communauté Flamande, Belgique). Le concept d’éducation interculturelle du nouveau-né Point d’Appui était
profondément influencé par son premier projet de recherche, une ethnographie de l’école. Ce concept s'appuyait
beaucoup sur des axiomes de la tradition ethnographique, particulièrement l’insistance sur la construction sociale
de la ‘réalité’. Dans cette approche, l’éducation interculturelle peut être différente dans chaque situation parce
que la diversité impliquée peut changer d’un contexte à l'autre. Je rejoignais l'équipe de recherche du Point
d’Appui au moment où il s’engageait dans quelques études d’évaluation. Cette opération nécessitait des
généralisations au-delà des contextes spécifiques. Le défi était alors de maintenir une perspective dynamique
dans une conception de recherche plus alignée à la tradition de recherche empirique-analytique que à l’approche
inductive. Nos concessions aux demandes des commanditaires ont sapé des principes essentiels de la tradition
ethnographique. Ces postulats seront élaborés dans la première section de l'article. Les chapitres suivants
représentent les tensions impliquées par l'application de ces principes dans la recherche d’évaluation.
L'évaluation suppose une définition claire soit une théorie sur le sujet de l'enquête. En plus, les commanditaires
ont exigé des généralisations, ayant au-delà du contexte si important pour l’ethnographie. Je maintiens que le
problème du rapport à la théorie (section 2) et le problème de la généralisation (section 4) ne sont pas propres à
la confrontation des cultures d'enquête dans notre recherche, mais sont plutôt des tensions inhérentes à la
tradition ethnographique. La section 3 présente une comparaison de deux études d’évaluation. La première
montre l'incompatibilité de paradigmes. Par contre, la deuxième est une tentative intéressante de réintégrer la
sensibilité au contexte dans la tradition empirique-analytique. Chacune des études d'évaluation discutées a mis
en pratique des techniques et des concepts ethnographiques. Néanmoins les fondements de la tradition ne sont
pas respectés. Des études qui s’inspirent par l’ethnographie devraient au moins prendre en considération les
différentes interprétations des acteurs du programme. Par contre, dans les études concernées, l’examination de
différentes perspectives ne servait uniquement à contrôler l’interprétation ‘correcte’ du programme. Dans ce but,
ils ont négligé la dynamique de modifications de significations personnelles et collectives. En plus, ils ont ignoré
le processus ouvert de l'éducation interculturelle.
Resumen
En 1995 el Centro de Educación Intercultural se creó en la Universidad de Gante (la parte holandesa de Bélgica).
Su primer proyecto de investigación, una etnografía escolar intensiva, influyó profundamente en el concepto de
educación intercultural del nuevo Centro. Este concepto se basarba en los supuestos de la tradición etnográfica,
especialmente en la inclinación de construcción social de la "realidad" actual. En esta perspectiva la educación
intercultural puede ser diferente en cada situación específica, porque el significado implícito de diversidad puede
cambiar de un contexto al otro. Me uní al Centro en el momento en que el equipo de investigación iniciaba la
evaluación global que requiria sobretodo generalizaciones de contexto. Entonces, el desafío era mantener una
perspectiva dinámica que tome en cuenta los contextos en un diseño de investigación más vinculado de la
tradición empírico-analítica que a la investigación naturalista. Hemos hecho concesiones sustanciales a las
exigencias de los comisionados de esas investigaciones, los que socavan supuestos esenciales de la tradición
etnográfica. Explicamos estos supuestos en la primera sección del artículo. Las siguientes secciones exploran las
tensiones en la aplicación de estos supuestos en la evaluación. Una evaluación supone la existencia de una
definición clara, o incluso una teoría sobre el tema de investigación. Además los comisionados exigieron
declaraciones generales, que van mucho más allá del contexto específico, central en la etnografía. Se puede
afirmar que esos problemas de teoría (sección 2) y de generalización (artículo 4) son inherentes a las tensiones
dentro de la tradición etnográfica. La sección 3 presenta una comparación de dos evaluaciones. La primera es
presentada como un ejemplo de la incompatibilidad entre paradigmas contrapuestos. Por otra parte, la segunda
muestra un intento interesante para reintroducir una alerta en el diseño de investigación hostil al contexto. Cada
uno de esos estudios de evaluación han aplicado técnicas y conceptos etnográficos. Eso no es suficiente para
respetar los supuestos fundamentales de la tradición etnográfica. Una investigación inspirada en esa tradición
debe al menos tener en cuenta las diferentes y cambiantes interpretaciones de los programas. Pero en estos
estudios las diferentes interpretaciones de las participantes fueron tomadas en cuenta, sólo para medir una
comprensión ‘correcta’ del programa. Como tales, los estudios de evaluación desatendieron la dinámica de
significados cambiantes en una realidad cultural diversa. Además abandonaron su propia definición de educación
intercultural como un proceso abierto.
The concept of ‘intercultural education’ has become part of the Flemish governmental school policy
since 1991. In the course of consecutive policy programs schools with a certain amount of pupils
originating from ethnic minority backgrounds or meeting (contested) criteria of disadvantage could
apply for extra support when implementing initiatives of intercultural education. As the commonly
held interpretation of the ‘freedom of education’ in Belgium restricts the impact of government policy,
the meaning of ‘intercultural education’ was delegated to the autonomy of the local school practice
(mediated by the powerful educational networks). In 1995 the Centre for Intercultural Education
(further shortened as CIE) was founded at the University Of Ghent to supply research, in-service
teacher training and the development of materials1. As a consequence of the ‘freedom of education’
the Centre functions as a mediator between the policy of the Flemish government and the local
autonomy of schools, between mainstreaming the different local concepts of intercultural education
and giving support to local practice.
Crucial to the approach of the CIE was the very first research project, an intensive school ethnography
undertaken under the supervision of Ruth Soenen (1999, 2002). The starting point of this research was
explicitly not a ‘didactic’ or ‘psychological’ approach, but the classroom as a ‘cultural activity’. The
purpose was to present an account of every day life with a primary interest in social interaction.
Although explicitly not the purpose, the study provided the Centre with a theory on intercultural
education. In this approach intercultural education should start by observing the existing different
meanings of diversity in order to get rid of fixed ideas and stereotypes and to explore the dynamic of
shifting meaning in a multilayered social/cultural reality. Existing patterns and recurring insistence in
certain behaviour should be, in one way or another, enlarged but equally starting from the already
existing meanings of diversity. The ‘enlargement’ is open-ended. As people move through very
1
The Centre recently changed its name to Centre for Diversity and Learning.
different social contexts (or numerous situation-bound cultures) and these contexts change
continuously it is impossible to predict ‘best’ ways of adaptation. This approach clearly reflects some
features of the ethnographic tradition, as I will develop in chapter 1.
The consequence of this approach created major problems in its application to in-service training. As
embedded in the a priori conception of education, teachers want to know ‘where’ to go to and
especially ‘how’. In a context-related conception of intercultural education ‘how’ is different in each
specific situation and ‘what’ is insecure. Already from the start of the CIE this tension inflicted the
relationship between “researchers” and “trainers”, the latter demanding for recommendations overall
contexts. I joined the CIE as a researcher at another crucial moment, the engagement of the research
team in evaluation research, posing specific challenges that I will explore in chapter 2. Once again the
tension between context-related diversity and evaluation statements requiring overall context
generalisations became prominent.
How to maintain a dynamic perspective when forced to ‘rank’ practices and asked for a state of the art
of the implementation of intercultural education in Flanders? Is it possible to combine such opposing
propositions as an ideographic starting point but a nomothetic inquiry aim? How to reconcile
reconstruction with prediction, explanation with understanding? These questions reflect old
discussions in the qualitative and especially ethnographic tradition. If we assign to ‘the ethnographic
tradition’ the status of a paradigm, as some authors tend to do, the whole discussion brings us back to
the question of reconciliation of paradigms (Fetterman, 1988; Patton, 1988; Guba & Lincoln, 1994).
Could we apply concepts derived from one (inductive-naturalistic) paradigm to a research design
embedded in another (positivistic-deductive) paradigm? Is it possible to embrace ontological and
epistemological propositions of one paradigm but to use the research procedure of the other? The CIE
experience illustrates that methodological tensions in qualitative research strongly coincides with the
conceptual ones (chapter 3). Whereas the analysis of the school ethnography broadened the concept of
diversity to the shifting meanings of differences and similarities, already in the first evaluation study
this conception was narrowed again to observational behaviour.
1. The ethnographic tradition
At the start of its research activities the CIE clearly situated itself in the ethnographic tradition.
However, it is quite precarious to define this tradition. Ethnography escapes ready summary
definitions and is subject to much constraints and contradictions and a multiplicity of different
interpretations (Atkinson, et al.: p. 1). For the case of this contribution I will define a study undertaken
in an ethnographic spirit to meet –at the same time- three criteria:
(1) First-hand exploration of daily practices;
(2) The use of a specific inductive, interactive and circular methodology;
(3) The inclination to the social construction of ‘reality’ and the importance of social context.
Each of these features can easily be recognised in the first study of the CIE: a school ethnography
undertaken by Ruth Soenen.
School ethnographic research in primary education (1995-1998)
Ruth Soenen engaged in the exploration of the everyday interaction in the environment of teachers and
pupils (2002, p. 78). Fieldwork in three primary schools was undertaken from 1995 till 1998 with a
main emphasis on observations (mainly classroom observations but researchers were also present at
the playground, on excursions, etc.) combined with interviews and informal talks. The purpose was to
present an account of every day life in 4th and 6th grade (age of pupils between 9-14 year) in 2 primary
schools characterised by an ethnic mix of the school population (fieldwork involved: one year). A 3rd
primary school with an ethnic mix below 30% was included in the research project as a contrast school
(fieldwork involved: 4 months). The focus of interest was the social interaction and the ways in which
environmental contexts impose restraints on this interaction.
1.1 First-hand exploration of daily practices
The first criterion refers to the use of specific methods of data collection: participant observation, open
ended interviews, informal talks. Yet, the use of these methods does not suffice to call a study
“ethnographic”. Atkinson et al. complain that “all too often authors and researchers are talking about
the conduct of in-depth interviews (…) divorced from contexts of social action.(…) These are often
important ways of gaining principled understandings of social life and personal experience, but
should not necessarily be equated with ethnographic research.” (p. 5). Ethnographic research,
Atkinson et al. claim, should be firmly rooted in the first-hand exploration of research settings. In this
respect our first criterion does not only refer to techniques, but also implies the environment of
investigation: human social activities should be studies in their naturally occurring circumstances
(Dorr-Bremme, 2001: p. 4).
1.2 An inductive, interactive and circular methodology
First-hand exploration of daily practices in itself is once again not a sufficient condition in the
demarcation of an ethnographic study. Also implied is a specific methodology, embedded in an
“ethnographic epistemology”. As Guba & Lincoln state:
“The hammer may be the method, but using it in the service of carpentry is an instance of
methodology. One can mix and match, or blend, hammers, saws, wrenches, levels, and the
like, but one cannot mix and match or blend carpentry with, say, plumbing.” (1988: p. 91).
Most Anglo-Saxon literature refers to ethnography as a “theoretical tradition” (Patton, 2002) or an
“inquiry paradigm” commonly equated to the paradigm(s) of naturalistic research (e.g. in Cohen,
Manion & Morrison, 2000). That doesn’t solve our problem of demarcation. A paradigm includes
several axioms, so researchers may share some of these assumptions but reject others. Guba & Lincoln
insisted on several occasions the non-miscibility of methodologies as methodologies are rooted in
paradigms. Still a paradigm seems to be ‘larger’ than the methodology. One can for instance adopt
(ontological) premises about the indeterminism and dynamics of cultural life but at the same time
indulge in a rather conventional empirical-analytical semi-closed, linear oriented research design. This
is exactly the point I want to make in contrasting the CIE’s starting school ethnography with the
following evaluation designs. However, for the sake of argument, I will attribute a specific
methodology as an essential element of ‘ethnography’. Essential to the ethnographic approach is its
inductive, interactive and circular collection of data (Lecompte & Schensul, 1999: p. 9-21). As Ruth
Soenen clarifies this approach (2002: p. 79):
“In setting up the design for the research, it was not possible to define a number of variables
in advance that could then subsequently be studied in context. I did not, therefore, reduce
beforehand. The variables were generated through the processes of interaction. Therefore, in
the first phase, there were no sharply defined hypotheses. As many data as possible were
gathered and noted. (…) One has to start out with a broad interest in order to maximize the
capacity to learn. Along the way, the broad interest can be refined into the form of a number
of preliminary hypotheses that are constantly confronted by everyday life.”
This quotation is almost a mirror of Guba & Lincoln’s description of the methodology of naturalistic
inquiry (1988, p. 105). “Naturalists are unwilling to assume that they know enough about the
time/context complex a priori to know what questions to ask. (…) But naturalists typically enter the
context as learners, not claiming to know beforehand what is salient.”
1.3. Inclination to the social construction of ‘reality’
Ethnographers share common assumptions about the social construction of reality, the insistence on
human beings as attributing (different) meanings to reality, the importance of context, etc. (fully
elaborated in for instance Cohen, Manion, Morrison, 2000: p. 137-140). However, once again, one
may approach this ‘reality’ in many different ways. In this respect Hammersley observes an increasing
fragmentation of the qualitative research community (2002: p. 7) with two ends on the spectrum:
“post-positivism” on the one hand and “post-modernism” or “constructionism” on the other. To avoid
possible misunderstandings, I should make clear that Hammersley’s notion of “post-positivism”
contradicts with Guba & Lincoln. In their elaboration of competing paradigms the latter attribute
explanation (prediction and control) as the inquiry aim of both positivism and post-positivism. In their
view post-positivism refers to that position where knowledge consists of non-falsified hypotheses that
can be regarded as probable facts and laws (1994: p. 113). However, in the sense as Hammersley is
confronting post-positivism with post-modernism and as we are balancing on a continuum, I suggest
we could also include researchers aiming at understanding and reconstruction to belong to the ‘postpositivist’ side. Post-positivists in this interpretation still believe in ‘facts’, but these ‘facts’ have a
totally different nature than perceived by positivists. The goal of the observer should be to obtain data
that correspond with reality, even when we may have doubt about the possibility to ever reach that
goal. This ‘reality’ is not a reality as perceived in the positivist naïve realist ontology, but a dynamic
social reality. Therefore, we have to proceed with research procedures to enter this reality from the
inside. This is the position Soenen takes.
“I saw ethnographic research and qualitative research in general as the method par
excellence to get closer to the true nature of social phenomena.” (2002: p. 77)
Soenen, as most ethnographers, is not interested in explanation in the sense of prediction and control.
Still, she remains interested in the ‘facts’, in the reproduction of ‘life-as-it-is’. If she is not aiming at
explanation any more, if she is refusing to generate hypotheses before the fieldwork, this is mainly
because explanations offer bad reproductions of ‘life-as-it-is’. To know this life-as-it-is we should take
part in this life. Although the interaction between researcher and the studied subjects forms a potential
source of bias, we have no other possibility to engage in social reality in order to know this social
reality. However, the researcher should take distance from this interaction. S/he should represent the
different Emic perspectives, but in the same movement relate to the subjects and their own subjectivity
with a view from nowhere.
“It was important to check to what extent the data obtained corresponded with reality, whether
they were true or false in their own situation.” (Soenen, 2002: p. 79, my emphasis)
Post-positivists in this sense may recognise the multiple realities involved in the same context and
situation, as constructionists do, but believe that we are capable of representing these multiple realities
in a global overview. Moreover, this global overview should be as ‘neutral’ as possible, with the
primacy of validity as a criterion in assessing research findings.
Hammersley observes two challenges to this concept of validity as representation of the relevant facts
of the matter. The first emerges from what he calls “activism”, the idea that researchers can and
should play an active and central role in the relation to political and other kinds of practice. In his view
there is a tendency whereby activists begin to argue that the aim of research must be to produce
knowledge in such a way that is contributes to realizing some desirable practical goal (2002: p. 10-11).
The challenge to validity then is that it is no longer a matter of correspondence with ‘life-as-it-is’, but
a matter of what has desired implications in terms of the practical, political or ethical project. The
other challenge to the post-positivist conception is what he refers to as “constructionism”. In this
perspective all accounts of the world, including researcher’s accounts, are constructions which in some
sense create the things described. Data are no reflection on reality, but form a constituent part of it.
The challenge to the validity concept is clear. Any claim of ethnographers to produce accounts whose
validity is superior to that of others will be dismissed as a spurious appeal to authority. As a
consequence the kind of writing encouraged by many constructionists are those that leave open the
interpretation of data to readers, or which continually subvert their own claims to providing valid
knowledge (Hammersley, 2002: p. 12-13).
Soenen claims to refer to both a ‘factist’ perspective as to a ‘specimen’ perspective, the latter
corresponding to the constructionist claim about the ongoing social construction of reality. However, I
do believe that she is not fully taking into account the consequences of the ‘specimen’ perspective.
People construe very different personal worlds of meaning of the same situation, she argues, and we
should concentrate on ‘how’ these multiple realities are constructed. In my view this is just an
extension of the ‘factist’ perspective, whereas the ‘facts’ to be found are the specific mechanisms, the
specific rules ‘that construct the different worlds of meaning. We should in the analysis include the
interaction of the researcher with his subjects, says Soenen (2002, p. 81). But the reason for doing this
is still embedded in a ‘factist’ perspective: if we are to describe the ‘facts’ of social reality, then
interaction with the researcher is part of this and can not be left behind.
2. Theory in doing ethnography
When I joined the CIE it engaged in the first of a series of qualitative evaluation studies related to the
concept of intercultural education. All these studies involved, in one way or another, an assessment of
the level of implementation of intercultural education. Moreover, all three studies were embedded in a
larger framework of evaluation in cooperation with other research centres. These frameworks of
evaluation all included a research design more familiar to the empirical-analytical tradition than to
naturalistic inquiry. As such the researchers were confronted with two main challenges.
(1) Assessment supposes –at least in the empirical-analytical tradition- a clear definition of, if not a
theory about intercultural education. This concept then had to be made operational in order to
develop evaluation criteria. I will develop this problem in this chapter.
(2) The problem of generalisation. On the base of a sample we had to make general statements about
a larger population. Observation and interviewing some teachers led to statements about the whole
school; the analysis of some schools led to statements about all schools involved in a specific
programme. In the empirical-analytical tradition this problem is ‘solved’ with significance tests.
But could we on the basis of qualitative research whatsoever make statements about a larger
population than the subjects involved? I will come back to this in chapter 4.
2.1. Theory in an ethnographic tradition
The ambition of ethnography is not to build grand theories. Social reality is constantly and rapidly
changing, as humans produce and reproduce insistently different meanings. As such, reality escapes
transcendence in conceptualisation and theory building. The concern of ethnography then is rather the
“authenticity of contextual understanding” (Williams, 2002: p.129). Although concepts and theories
are not at all excluded, they should never be presented as representations of general laws, but rather as
working hypotheses describing time- and context bound individual cases.
In the naturalistic inquiry tradition some researchers even reject theory at a starting point of
investigation. The inquiry is by some researchers even considered to be “theory-free”, exclusively
based on the data that emerge as on-site investigation proceeds (Dorr-Bremme, 2002: p. 5). This is
however a misleading conception. As Dorr-Bremme argues, even when adopting the inductive and
circular research cycle, theory is involved from the very beginning.
“… investigators cannot escape choices about how to focus inquiry, what to include as data,
and how to describe what you have seen. They must decide, for example, what sites to visit,
which setting within each site to focus on, when to be there, and how often and for how long.
(…) Similarly, ethnographers or naturalistic-qualitative evaluators must repeatedly make
choices throughout the course of inquiry about whom to interview formally and whom to talk
to informally, what questions to ask, and how and when to ask them. Then they have to decide
which words and other behaviours, of all those that occur each moment in each conversation,
deserve a place in the study record. Furthermore, as inquiry proceeds, they must constantly
determine which terms, behaviours, and so forth that they encounter on site can be safely be
treated as unproblematic, their purposes and meanings sufficiently clear, and which should be
treated as problematic, their purposes and meanings requiring further inquiry.” (op cit: p. 5)
From the very outset of inquiry, these decisions are made constantly on the basis of the investigator’s
perspective on what is important. As such:
“The problem is not the elimination of ‘bias’ in description, for all description is done in
terms of a point of view. Rather, the problem is the selection of ‘bias’, or theoretical frame,
appropriate to the research problem at hand.”(op cit: p. 6)
The insistence on this theoretical frame doesn’t contradict the inductive, interactive and circular
methodology. Instead, interaction with this theory is part of the methodology and the awareness that
our glasses are inevitable coloured should warn us for an all too naïve epistemology.
2.2 Intercultural education with reference to the ethnographic tradition
The analysis of the school ethnography didn’t provide a theory on intercultural education as such.
However in retrospective, Soenen's assumptions resemble a great deal of theoretical assumptions that
found their way in CIE’s definition of intercultural education: “learning to manage social and cultural
diversity in an active and efficient manner” whereas each of these words gets a specific interpretation
(see Verlot et al, 2000).
Social and cultural diversity in this definition should be taken in a broad sense, involving similarities
as well as dissimilarities. Important to the implementation of intercultural education is not so much the
actual feature of diversity (colour, religion, age, interest, ethnic background, gender, …), as well the
meaning these features get in specific contexts. Intercultural education was introduced in Flanders as a
practice of dealing with migrant pupils in the classroom. Not surprisingly, a lot of schools still use an
ethnic concept of intercultural education. The CIE strongly objected to such a reduction of diversity.
Pupils embody not only their ethnic background. They act and react, create as well as reproduce
different types of ‘cultural’ behaviour referring to their belonging to different cultural communities. In
her analysis of the school ethnographic research Soenen uses the idea of modes of interaction
(borrowed from McLaren, 1993) to clarify the thesis of children’s acting and reacting on very different
frameworks of reference. A mode of interaction is defined as the totality of behavioural and interaction
patterns developed by the children themselves but referring to different group relations. Soenen
discerns at least three recurring modes of interaction (1999, 2002).
 The ‘child interaction mode’ refers to belonging to a certain family and entails ethnic, religious,
social-economic and/or family elements. Some of the elements are common with other pupils.
Others elements are not.
 When children meet some criteria induced by the school they act and react within the ‘pupil
interaction mode’, referring to behaviour common to all children in the same school. Ruth Soenen
describes this ‘pupil interaction mode’ as mainly meeting the daily routines in the classroom and
performing a rather passive and reserved line of conduct.
 The ‘youth interaction mode’ is developed in peer group relations. Soenen gives illustrations of
mutual interests (in exciting stories, in the private life of the teacher, in specific music and
fashion), of specific abilities (negotiation skills, for instance about the rules of a game or in
merchandising, ways to undermine the purposes of sanctions, the skill to combine different
activities) and specific style elements of communicating. The character of this kind of behaviour is
rather expanding and chaotic, especially when compared with the pupil interaction mode.
The occurrence of these modes of interaction changes in different environmental contexts, as well as
the specific patterns of behaviour involved. Soenen explores how in different schools the internal
hierarchy between modes of interaction could be almost completely altered. Her final analysis of the
school ethnography presents two vignettes: two contexts to illustrate the changing meanings of
diversity in these different contexts. The first context was nominated the context of the ‘Gauls’ as
referring to the atmosphere among the villagers in the comic Asterix: a vivid, but rather chaotic
environment with characters that continuously change coalitions but unite in the struggle with the
enemy. The contrasting context was referred to as the context of the ‘Managers’, as the pupils in this
school “a foreshadowing of future managers who put people in their place and who aspire to
perfection in their products and who compete with each other.” (Soenen, 2002: p. 85) Whereas in the
context of the ‘Gauls’ all three interaction modes recur, in the context of the ‘Managers’ different
ways of behaving are hardly seen at the school, as pupils reinforce themselves the importance of the
pupils interaction mode.
The crux in relation to ethnicity is her assertion on pupils creating their own hierarchy. Therefore, one
cannot say in advance which differences will be important (op cit, p. 89). In the investigated
classrooms in primary education the criteria for separating groups by the pupils themselves rarely
involved ethnic borders. In the context of the ‘Gauls’ the criteria to divide into subgroups were related
to the ‘youth mode of interaction’ (gender, status, etc). In this context pupils with an ethnic minority
background could for example excel among peers because of their (peer group related) communication
abilities. In the context of the ‘Managers’ the criteria for separating groups mainly referred to the
‘pupil mode of interaction’. Once again, ethnicity disappears in the background. Pupils with an ethnic
minority background could be widely accepted among their peers because of behaving as a model
pupil.
Culture then refers to a non-essentialist, dynamic concept of culture, mediated in interaction. As DorrBremme clarifies, the fundamental supposition of most ethnographers is that interaction of human
groups is socially organised according to a series of principles (a ‘culture’) for making sense of the
world and acting sensible in it (2002: p. 8). The dynamic aspect of culture refers at the one hand to
changes in the ‘content’ of these principles, as can be recognized in the rapidly changing ‘contents’ of
youth cultures. On the other hand the dynamic of culture is embedded in our taking part in numerous
situation-bound cultures. A “social group’s standards for sensibility and appropriately interpreting
phenomena, ascribing meaning and value and choosing actions can vary with features of the social
context or situation” (Dorr-Bremme, 2002: p. 9-10). One can easily recognise the modes of interaction
in this central argument. Within the context of the school pupils make ‘now’ and ‘next’ interpretations.
Within these alternatives proper to this context they could however also refer to their belonging to
various social groups. They are not only pupils, but also peers and members of a certain family, as
they are also involved in leisure activities or street life, etc.
If intercultural education refers to “dealing with diversity” this management includes (1) the
recognition (of shifting meaning) of diversity as ‘normal’ and (2) approaching diversity as a source of
learning. In other words: intercultural education is the management of the class and school life in such
a way that learning from each other becomes obvious. Soenen stresses very much the importance of
observation in intercultural education. “What our research proposed is not a delineated pedagogical
method, but rather an initial impetus to an observation instrument, in which it is important to observe
in the first place the here and now.” (Soenen, 2002: p. 89)
3. Clash of Inquiry Cultures?
Patton recommends qualitative researchers to be “methodologically flexible and situational
responsive” but the CIE’s trial to integrate ethnographic notions in a ‘traditional’ evaluation design
fully showed the difficulty of being “methodologically eclectic” (1988: p. 128). Both evaluation
studies described below were challenged by different inconsistencies when adopting assumptions from
the ethnographic tradition in an evaluation design more familiar to the empirical-analytical
quantitative tradition. In what follows I will contrast an example of an unfruitful clash (3.1) with an
intriguing effort to integrate ethnographic context-awareness in the competing research design (3.2).
The chapter will end with an exploration of the problem of generalization.
3.1 Clash
The tension between context-related diversity and evaluation statements overall contexts became
prominent at the start of the first evaluation study regarding the evaluation of the implementation of
the Non-Discrimination Treaty. This Treaty was signed in 1993 by all educational networks, labour
unions, confederations of parents and the Flemish government. The subscribers of the Treaty engaged
in “gentlemen’s agreements” between schools to avoid the concentration of children with an ethnic
minority background. The CIE got involved in the evaluation of the Treaty because of the second
engagement of the Treaty: the promise that schools would invest in the implementation of intercultural
education.
Evaluation of the implementation of the Non-Discrimination Treaty (1998-1999)
The study was undertaken by a coalition of three research centres, one investigating the actual level of
desegregation of pupils movements, a second analysing the functioning of the municipal NonDiscrimination Assemblies and finally the CIE describing the implementation of intercultural
education. All studies concentrated on 5 municipalities out of 37 where Non-Discrimination Treaty
resulted in a local agreement. Inge Pelemans made observations in 3 schools in each municipality in
the first (age of pupils between 6-8 year) and fifth grade (age of pupils between 10-13 year), combined
with interviews with the teachers of this particular grade and their principals (all done in one week of
fieldwork in each school). (Pelemans & Verlot, 1999)
The main challenge for this study was to construct a measuring-yard in order to make comparisons
between local situations, and between the research results of all involved research centres. On top of
this challenge the researcher had to meet very strict time arrangements whereby she had to restrict
fieldwork in every class to maximum one week. The research team developed a semi-structured
observation scheme based on the analysis and premises of Soenen with a main interest in the
occurrence of modes of interaction. Soenen presents these modes of interaction as “eye-openers” to the
diversity of meanings, insisting on the importance of the context. Pelemans however reduced the
notion of mode of interaction to observational behaviour, assigning a one-to-one relation between a
particular kind of behaviour and a mode of interaction, for instance: “laughing” in the classroom
would be coded in each situation as “youth interaction mode”. In this movement the ethnographic
insistence on context-relatedness was completely ignored by transferring “eye-openers” to a pre-coded
model of observation. Or in relation to the example: the study completely ignored the difference
between laughter in the context of the Gauls and laughter in the context of the managers, or laughter at
a moment the teacher deliberately makes a joke and laughter when the teacher expects complete
silence.
The clash continues. The final report of the study is full with statements as “the pupil mode of
interaction takes 76% of all modes of interaction”. The analysis was based on a very disputable
quantification of indexing categories. Within the NUD*IST software programme (current: NVivo) one
has the possibility to count the imported lines of observation that were coded within the same
category. The researcher in this case just counted all lines of observation she coded as “pupil mode of
interaction” and compared them with the lines coded as one of the other modes of interaction. She
neglected that already from the moment a researcher uses more words to describe a situation the
involved lines expand increasingly.
3.2 Possible reconciliation
In the preamble of the 2002 Act on Equal Opportunities (that replaced the program under concern) the
Education Department asked for an evaluation of the Educational Priority Policy. This policy was
aimed at target group pupils in primary and secondary education who because of social, economic or
cultural reasons have learning difficulties. Schools were granted additional financing for each target
group pupil based on mainly two conditions.
(1) A satisfactory level of presence of the target group in the school: principally pupils from a
different ethnic background.
(2) The establishment of an allocation plan, indicating how the school would spend the extra
resources. One of the central principles was the idea that the extra support should improve the
general quality of education in the whole school and should not only focus on extra (individual)
support of the target group. The allocation plan should therefore describe the actions taken in one
optional field of activity and 4 obligatory fields of action, which included the obligation to engage
in intercultural education.
Evaluation of the implementation of the Educational Priority Policy (2000-2001)
Also this study was undertaken by a coalition of two research centres from two different universities.
Whereas the CIE investigated the implementation of the 4 action fields of the Educational Priority
Policy in 20 primary schools (Hillewaere, 2001), the KU Leuven (HIVA) assessed the initial effects on
the learning progress of the target group in the same classes in the 4th and 5th grade (age of the pupils
between 9 and 12 year). The ambition of the research design was to provide an account about critical
success factors by comparing the analysis of both investigations (Schrijvers, et al, 2002). Katrijn
Hillewaere engaged in two weeks of observation and interviews in each class. With regard to the
action field of intercultural education she adapted the observation scheme from the study of Pelemans.
The semi-structured interviews were focused on the implementation of all action fields on the
Educational Priority Policy, therefore also providing the discourses of teachers concerning the relation
of intercultural education to the other action fields.
The assignment for this study was to account for critical success factors, stemming from the
confrontation of the analysis of the two involved research centres. In order to make a comparison
possible the study had to make a ranking of the involved schools according to their level of
implementation of the Educational Priority Policy, indicating which school was doing better than the
other. If we take the criteria elaborated in chapter 2 as proper marks of discrimination, one may very
much hesitate to call this an ethnographic approach. Still, there is an interesting attempt in this
research to include context-awareness in an evaluation design dominated by the empirical-analytical
paradigm. It is my conviction that this attempt could be fruitful in considerations about possible
reconciliation of the competing paradigms.
The research report itself unfortunately gives very little information about the analysis leading to a
final ranking of schools, so this account is based on my own involvement in the research team at that
moment. At first the analysis was very much inspired by the approach of Pelemans: a pre-defined
meaning assigned to the conduct of teachers and pupils. For example: if in the observed lesson the
teaching method mainly consisted in the teacher talking and the pupils listening, this conduct would be
ranked as one of the possible indications of a ‘bad’ implementation of intercultural education. The
rationale behind would be that the interaction between the pupils is not promoted. One problem with
this argument is that it reduces ‘interaction’ to face-to-face interaction with both parties actively
participating. The main problem however is that we don’t get a clue about the meaning of this
situation to both teacher and pupil. Do pupils feel oppressed, not daring to share their own experiences
and therefore enhancing the diversity in the classroom? Or do we witness a context of ‘Managers’
where pupils themselves reinforce the cliché of traditional classroom management? Or, quite contrary,
are we in a context of ‘Gauls’ and do we face one of these rare opportunities where the teacher
succeeds to break through an existing class climate of chaos?
Yet, as the researcher herself engaged in first-hand exploration of daily practices, she developed quite
some intuitions about different interpretations of the observed situations. When reviewing her first
analysis, counting the different fixed indications of the level of intercultural education, the result
showed in several cases counterintuitive. After the observations but before the analysis she would
define particular teachers as (very) good examples of promoting intercultural education. Yet, after the
initial analysis (based on counting), they only showed somewhere in the middle of the ranking.
Hillewaere consequently adapted the whole frame of analysis in order to meet her intuitions. The
frame became much more fine-meshed. In the given example she reconsidered the criteria of ranking,
now not only taking into account the teaching method, but also the way this teaching method was
used. A teacher directing the whole classroom interaction by mainly taking the word herself could still
promote diversity, for instance by making jokes or by referring to the interests of children, etc. The
adaption resulted in a complex system of balancing different indications of the level of intercultural
education. A teacher that used ‘traditional’ teaching methods could be in one context ranked as rather
‘bad’. Still, another teacher using the same methods in another context could rank as rather ‘good’ if
also taking in account several other indicators.
This whole revision of the initial frame of reference could be dealt with in two ways. Hardliners of the
empirical-analytical paradigm will consider this whole operation ‘illegal’. In their view intuitions have
no place once the research design is fixed. If the first analysis proves counterintuitive, then believe the
‘facts’ and reject the intuition. Quite the opposite reaction is to embrace the whole revision as an
upgrade of the validity of the study: the new frame of analysis covers the investigated concept of
intercultural education in a better way than the first scheme did. As the concept of intercultural
education includes the possibility of different manifestations, this diversity should be included in the
research design to ‘really’ measure what you want to measure.
However, what strikes my attention in relation to the topic of this article is the methodology. Although
the main frame of approach is situated in a deductive cycle, Hillewaere retakes elements from the
inductive, interactive and circular methodology to reframe the whole analysis. Although ‘ranking’
includes normative aspects transcending the manifold of interpretations of the situation, still this
ranking takes into account the context-relatedness of any interpretation.
4. The problem of generalization
Ethnography, at least in our demarcation in chapter 2, reconstructs daily life in a particular context
involving context-related and shifting meanings. Yet, the commissioners of the evaluation studies
required statements about the implementation of a programme overall contexts. This brings us to the
problem of generalization.
In practice, all evaluation studies made general statements, going far beyond the observation of the
particular individual teacher in his specific context. The researchers state for instance that
“intercultural education is in a shabby condition” (Verlot & Pelemans, 1999) or that “the
implementation of the concept is rather moderate” (Hillewaere, 2001). But, actually, so does the
school ethnography in its representation of two vignettes. It is unlikely that ethnography escapes the
problem of generalization. The presentation of an ethnographic study inevitably involves a radical
compression of data. Although these studies present “thick descriptions”, the selection of which
descriptions to include is not arbitrary, but commonly reveals more general patterns (or exemptions to
this pattern) as perceived by the researcher. There is off course an important ‘but’. In the ethnographic
tradition, these patters are likely to be bound to a given time and context. Soenen indeed reveals
recurring modes of interaction (a general pattern), the important point she is making is exactly the
changing hierarchy between these modes of interaction, given the time and the context. The basic
assumption of the study is “vibrant symbolic creativity in every day life” (Soenen, 2002: p. 87). If the
study presents general statements, these statements refer to the multiplicity of social reality.
The issue is not that the evaluation studies made generalizations, but rather that a particular kind of
generalization is made that reduces the multiplicity of social reality central to the ethnographic
tradition. The construction of a measuring-yard in the study of Pelemans for instance negated this
multiplicity, by assigning only one possible definition to the situation. It is this movement that
separates the evaluation studies from the ethnographic tradition that emphasises the fundamental
“inherent indeterminateness in the life world” (Williams, 2002: p. 130).
Taken this fundamental indeterminateness then, what kind of generalization is possible? Williams
discerns two positions. In the position of ‘relativist interpretivism’ (op cit: p. 137) we can only try to
reconstruct the different interpretations in a specific context at a specific time. Taken from this
position the validity of the evaluation studies is low, as they only reflect the interpretation of the
researcher (or commissioner) of the situation and as the claim to statements overall contexts becomes
absurd. The relativist interpretivism however ignores an important characteristic of the social world:
its cultural consistency. Stressing this cultural consistency leads to the second position, the position
that claims that “moderatum generalizations” are possible.
“To claim the existence of cultural consistency commits one to at least a minimal form of realism
of the kind advocated by Blumer, whereby our accounts of the world are regarded as substantially
reliable, yet incomplete and erroneous. We are attempting to describe the reality of the people we
investigate. That the accounts produced are more than a ‘story’ or a ‘text’ is verifiable by those
investigated, but it is more than this. Those ‘realities’ as experienced are often the outcomes of
processes, the evidence of structures existing beyond the individuals investigated.” (op cit: p. 138)
This position still leaves open the question what kind of “moderatum generalizations” are possible. We
might accept the modes of interaction, but do we also accept the measuring-yards from the evaluation
studies, be it in the version of Pelemans or the fine-meshed variation of Hillewaere?
5. Applying ethnography in evaluation: without rhyme or reason?
Each of discussed evaluation studies applied ethnographic techniques and concepts. According to
Fetterman this is a sufficient condition to label the studies as ethnographic educational evaluation
(1988, p. 45). Indeed, one may still recognise in the studies features reminiscent of the ethnographic
tradition. Although the evaluation design was embedded in a deductive and nomothetic paradigm, the
researchers clearly invested in the first-hand exploration of daily practices, even with a very strict time
schedule. Although the observation scheme predefined the interpretation, the categories of
observations were taken from the school ethnography. And although the studies didn’t take into
account the informal talks with teachers of pupils, the conduct of formal interviews was guided by the
ethnographic tradition, trying to make the conversation as common as possible, avoiding direct
questions that would beforehand impose the meaning the researcher is assigning to a specific concept.
And at a level of analysis the study of Hillewaere offers an interesting attempt to re-introduce contextawareness in a context-hostile research design.
Yet the studies made important concessions to the demands of commissioners, undermining essential
assumptions from the ethnographic tradition. The first principle for the design and conduct of
evaluation fieldwork, following Dorr-Bremme, is “to treat the program’s definition and boundaries as
problematic” (2002: pp. 13).
“It should consider participants’ interpretations of “the program” as a central manner
during inquiry. [The primary interest of ethnographic evaluators] should not be in whether
participants at a particular site arrived at a ‘correct’ understanding of the program. Instead,
evaluators interested in participants’ realities should be concerned with how the
interpretation(s) apparent at this site have been achieved and how these interpretations
function in the program’s enactment. (…) And recognizing the perpetual interdependence
between participants’ ongoing interpretations and actions, ethnographic evaluators should
consider participants’ interpretations of the program as likely to be dynamic rather than
static.” (Dorr-Bremme, 2002: p. 13-14)
Instead of treating the definitions of the programs to evaluate as problematic, these definitions clearly
informed the construction of the measuring-yard. If different interpretations of participants were taken
into account, than only to measure to which account they met a ‘correct’ understanding of the
program. The time for fieldwork involved left no room at all to even consider an analysis of changing
participants’ interpretations. As such the evaluation studies neglect the dynamic of shifting meanings
in a multi layered cultural reality. And they even negate the own definition of intercultural education
as an open ended process.
Patton argues that pragmatism can overcome seemingly logical contradictions (1988: p. 127). He
believes that in practical evaluation situations “the evaluator can view the same data from the
perspective of each paradigm, and help adherents of either paradigm interpret data in more than one
way” (op cit). I’m not convinced the CIE succeeded in the “creative practice” Patton defends. But as a
former participant in the CIE’s research team I realize in retrospect that in the confrontation with
commissioners or associated research centres that maybe the methods, but almost never the
methodology or paradigms included were topics in debating the research designs.
References
Atkinson, P. et al. (2001). Editorial Introduction. In: Handbook of Ethnography (pp. 1-7). London:
Sage.
Cohen, L., Mansion, L. & Morrison, K. (2000). Research Methods in Education. Fifth edition.
London: RoutledgeFalmer).
Dorr-Bremme, D. W. (2001). Ethnographic Evaluation: A Theory and Method. In: Denzin, N.K. &
Lincoln, Y.S. (eds.) The American Tradition in Qualitative Research. Volume IV (pp. 3-28). London/
Sage. (Sourced Educational Evaluation and Policy Analysis, vol. 7, no. 1, 1985, pp. 65-83).
Fetterman, D. M. (1988). Ethnographic Educational Evaluation. In D. M. Fetterman (ed.) Qualitative
Approaches to Evaluation in Education (pp. 45-67). New York: Praeger.
Guba, E. G. & Lincoln, Y. S. (1988). Do Inquiry Paradigms Imply Inquiry Methodologies? In: D. M.
Fetterman (ed.) Qualitative Approaches to Evaluation in Education (pp. 89-115). New York: Praeger.
Guba, E.G. & Lincoln, Y.S. (1994). Competing Paradigms in Qualitative Research. In: N. Denzin &
Y. Lincoln (eds.) Handbook of Qualitative Research (pp. 105-117). London: Sage.
Green, J. (1994) Qualitative Program Evaluation. In: N. Denzin & Y. Lincoln (eds.) Handbook of
Qualitative Research (pp. 530-544). London: Sage.
Hammersley, M. (1990). Reading ethnographic research. A critical guide. New York: Longman.
Hammersley, M. (2002). Ethnography and the Disputes over Validity. In: G. Walford (Ed.) Debates
and Developments in Ethnographic Method (pp. 7-22). Oxford: JAI Press.
Hillewaere, K. (2001). Evaluatieonderzoek van het onderwijsbeleid ten aanzien van etnische
minderheden in het lager onderwijs. [Evaluation of the implementation of the educational priority
policy towards ethnic minority pupils in primary education] Universiteit Gent: Steunpunt
Intercultureel Onderwijs.
Lecompte, M.D. & Schensul, J.J. (1999). Designing and conducting ethnographic research. Walnut
Creek: Altamira.
McLaren, P. (1993). Schooling as a ritual performance. London: Sage.
Patton, M. Q. (1988). Paradigms and Pragmatism. In: D. M. Fetterman (ed.) Qualitative Approaches to
Evaluation in Education (pp. 116-137). New York: Praeger.
Patton, M.Q. (2002). Qualitative Research and Evaluation Methods. 3rd Edition. Thousand Oaks
(CA): Sage.
Pelemans, I. & Verlot, M. (1999). Evaluatie van het non-discriminatiebeleid. Deelrapport
interculturalisering van onderwijs in scholen [Evaluation of the non-discrimination policy. Partial
report on intercultural education] Universiteit Gent: Steunpunt Intercultureel Onderwijs.
Schrijvers, E., Hillewaere, K., Van de Velde V. & Verlot, M. (2002). Evaluatieonderzoek van het
onderwijsvoorrangsbeleid, OBPWO 99.15: samenvatting en beleidsaanbevelingen. [Evaluation of the
educational priority policy: summary and policy recommendations] KU Leuven: HIVA.
Soenen, R. (1999). Over Galliërs en managers. Bouwstenen voor intercultureel leren. [About Gauls
and Managers. Materials for Intercultural Education] Universiteit Gent: Steunpunt Intercultureel
Onderwijs.
Soenen, R. (2002). Diversity as a Tool for Ethnography. From a Critical Child to an Ethnographer
with Little Patience. In: G. Walford (Ed.) Debates and Developments in Ethnographic Method (pp.
73-94). Oxford: JAI Press.
Verlot, M., Sierens, S., Soenen, R. & Suijs, S. (2000). Intercultureel onderwijs. Leren in
verscheidenheid. [Intercultural education. Learning in diversity] Universiteit Gent: Steunpunt
Intercultureel Onderwijs.
Williams, M. (2002). Generalization in Interpretive Research. In: T. May (ed.) Qualitative Research
in Action (pp. 125-143). London: Sage.