Teaching advanced research and
scholarship skills to postgraduate
social science students
Paper presented at the 6th Leeds Learning and
Teaching Conference,
January 8th 2009
Linda Evans,
School of Education
What do we mean by ‘advanced’
research and scholarship skills?



‘advanced’ is a relative term
‘skills that research students and, indeed,
some academics often find difficult and
challenging’ (Evans, 2002)
The basis of my evaluation:


my teaching/supervision experience
my experience of reviewing submitted bids and
journal articles - and reading published ones!
Evans, L. (2002) Reflective Practice in Educational Research:
developing advanced skills, London, Continuum
What specific advanced skills?



Categorisation/classification
Conceptual clarity and definitional
precision
Formulating theory/theoretical
perspectives and propositions


theoretical models
Critical analysis

We ask for it, but seldom teach it.
A MA student’s attempt at critical
analysis:
an uncorrected extract from a draft essay
Evan’s comment that ‘amongst teachers who respected it and
responded well to it, ‘extended’ professionality in headteachers was an
effective motivator.’ (Evans, 1998, p130) struck a particular chord with
me because I am experiencing a greater sense of motivation since
changing schools in September. My new headteacher constantly
encourages ongoing reflection on individual practice with a view to
ensuring that each child reaches their potential. I am experiencing a
much greater sense of motivation as her professionality is very much in
tune with my own. Although my last headteacher at my previous school
was more of an extended professional than his predecessor, he was
unable to filter out his extended professionality to the rest of the school
due to his poor delegation skills: he wasted valuable time dealing with
things others could easily have managed. At my new school, from the
leadership team to the heads of department, the headteacher’s
professionality is clear. As a consequence the extended professionals at
my new school are more motivated than those with a similar degree of
professionality at my previous school.
‘Teaching’ critical analysis:
the rationale






Students generally have little or no conception of what
is meant by ‘critical analysis’.
They often underestimate the depth and degree of
originality that we are, ideally, hoping to see in their
work.
They consider themselves to be incorporating critical
analysis when, in fact, they are merely offering
superficial comments or observations.
They usually are able to identify critical analysis when
shown work that includes examples of it.
They naturally find it extremely challenging and difficult
to do.
They derive a tremendous sense of achievement from
having managed to do it well.
What do we mean by
‘critical analysis’?








questioning what you read in books and articles
finding flaws in what you read in books and articles
identifying contradictions in arguments and/or theoretical
perspectives
thinking of ways in which arguments and/or theoretical
perspectives may be enhanced or developed further
explaining intelligently why you approve of, or support, a
particular argument or theoretical perspective
explaining intelligently why you disapprove of, or have
reservations about, a particular argument or theoretical
perspective
originality of thought – your own ideas
developing sound, rational argument and/or discussion
A teaching tool:
identifying examples of critical analysis
On a one-to-one basis, using the student’s
own draft essays as a resource.
In group teaching:
 Using extracts from past students’ essays
 Using extracts from journal articles
 Using extracts from my own work, to allow
me to explain my thinking and thought
processes.

The Literature Review
Purpose of the literature review:
 to provide background and contextual
information against which to set your
argument
 to fuel your argument
 The purpose is not to show the tutor
how much you’ve read!
Reviewing the Material






The literature is a tool.
Use it to support your argument.
Use it to provide evidence of perspectives that
you can argue against.
Use conflicting views and contradictions to
develop discussion.
Draw on a reasonably wide range of authors and
texts.
Paraphrased references are as good as – and
often better than – quotes.
Using the Literature as a Tool
Do not simply present information found in the literature.
 Comment on it thoughtfully and intelligently.
 Indicate whether you agree with it:










to what extent;
why;
with what reservations or qualifications.
Argue against it.
Support it – but give your reasons for doing so.
Suggest modifications to it - but give your reasons for
doing so.
Pick holes in it.
Compare it with your personal experience.
Compare it with conflicting views/evidence/information.
Analyse it critically.
Riddell, S. (1989) Exploiting the Exploited? The Ethics of Feminist
Educational Research, in R.G Burgess (ed.) The Ethics of Educational
Research, Lewes, Falmer, pp. 77-99
After talking to one mother and father for nearly an hour and a half, I was
trying to bring the interview to a close with what I thought was a fairly
innocuous question. I asked:
‘Are you generally happy with the education Stephen’s getting?’
Mr Gammage, a policeman, replied:
‘Well apart from this bloody irresponsible strike that’s been going on…’
Instead of bringing the interview to a calm conclusion, this resulted in a half
hour’s denunciation of the entire teaching profession. Analyzing how I
responded to male attempts to establish power in the interview, I found that
I generally backed off from conflict, and often found it difficult to probe for
more detailed answers when what they said was very brief. Although I did
not probe the women’s responses either, they certainly offered me a far
more intimate view of themselves. Of the three fathers whom I interviewed
by themselves, one blocked my questions completely and gave yes/no
responses to everything, and another answered very briefly. The
headmasters whom I interviewed also provided proficient examples of
school politics in operation, spending a very long time avoiding my
questions.
From: Evans, L. (2002) Reflective Practice in
Educational Research: developing advanced skills,
London, Continuum: (Chapter 7: Telling it as it Is)
As a woman – and a feminist – I feel able to question with impunity that may well be
denied a man the researcher’s conclusion that these reported incidents demonstrate ‘male
attempts to establish power in the interview’. My immediate response to reading this and
similar passages in Riddell’s paper was to categorise them as evidence of partisanship and
of the author’s having a feminist chip on her shoulder. Certainly, that the researcher’s
experiences reflect male dominance – or attempts to dominate - is one possible
interpretation, but it is by no means the only interpretation. Ascribing motives to others particularly in the absence of their own accounts and explanations for them – is an
extremely inexact science that, at best, may be described as conjecture and, at worst, as
arrogant assumption. It is highly susceptible to misinterpretation that leads to the
generation of spurious conclusions. We are treading on thin ice if we subsume our
unsubstantiated interpretations of the motives behind the nature of their responses within
our data. This involves stepping over the line - albeit a thin one - that separates the
manner in which data are collected and provided and the data themselves; it blurs the
distinction between process and product. Based on the evidence available in her paper, and
bearing in mind that, like any paper, this is unable to provide us with the full picture, I
formulated the impression that the author is too ready to ascribe sexist or stereotypical
gender-related motives to her research subjects. Her male interviewees seem unable to put
a foot right: if they are loquacious and responsive she evidently interprets their responses
as domineering, and their reticence is evidently interpreted as attempts to sabotage the
interview through non-cooperation. This reflects what I interpret as her blinkered
viewpoint, and in a later section I examine ways in which she might have analysed and
presented her data in a more balanced way.
From: Evans, L. (2002) Reflective Practice in
Educational Research: developing advanced skills,
London, Continuum: (Chapter 7: Telling it as it Is)
Riddell’s analysis contrasts sharply with Luttrell’s (2000,
pp.513-514) description of how what was effectively the
removal of her blinkers allowed her to see and re-analyse
her research data in a new light:
Identifying my reluctance to deal with strong emotions and
mixed feelings about mothers – what could be called countertransference in my fieldwork relationships – marked a major
breakthrough in my research process. I again returned to all the
interview material, and through this recursive process I
discovered a range of maternal images and mixed feelings that
the women had expressed, but that I had minimized in my
analysis of the links between structure, culture and agency.
From: Evans, L. (2002) Reflective Practice in
Educational Research: developing advanced skills,
London, Continuum: (Chapter 7: Telling it as it Is)
Riddell’s interpretation of her male interviewees’
behaviour would be justifiable if she had applied
appropriate controls to, or at least consideration
of, other variables together with incorporation of
consideration of evidence that supported or
conflicted with her interpretation before
systematically assessing rival explanations. She
presents no evidence of having adopted such a
rigorous approach to her analysis.
From: Evans, L. (2002) Reflective Practice in
Educational Research: developing advanced skills,
London, Continuum: (Chapter 7: Telling it as it Is)
Yet I have to acknowledge another side to the arguments I have presented
within my criticism of Riddell’s analysis.
In his outline explanation of consensus theory Bridges (1999, p. 606) warns:
Philosophers have tended to be sceptical of consensus theory as an account
of what it means for a belief to be true…
Researchers ought perhaps to share especially in this scepticism of
consensus, since it is, arguably, their particular function as intellectual
citizens to challenge the easy and self-perpetuating consensus that society
creates for itself. At the very least they have to observe that what any group
of people believe or agree to be true may rest on, for example, unexamined
tradition, the hegemony of a dominant class, the suppression or selfcensorship of dissenting opinion or collective hysteria – all the things that
intellectuals and researchers have traditionally been expected to subvert.
Applying this consideration to my criticism of Riddell’s work, I am forced to
accept the possibility that, in presenting my interpretation of her analysis as the
reflection of a blinkered viewpoint, and my own as reflecting the more
consensual form of truth, I may have fallen into the trap, to which Bridges refers,
of perpetuating a consensus that fails to question - because it fails to see – the
hegemony of one or more specific societal groups. In a sense, moreover, I seem
to be trapped in a ‘Catch 22’-type situation; by contending that Riddell’s
viewpoint is blinkered does not my own viewpoint then risk becoming so?
Morale, Job Satisfaction and Motivation:
Research Questions

What factors influence morale?

What factors influence job satisfaction?

What factors influence motivation?

Activity: examine the raw research data presented
as example quotes, in order to find answers to the
above research questions.
Conceptual clarity and
definitional precision



I do not tell students in advance that this is
my teaching focus.
Students better appreciate the need for it and
the potential difficulties created by its
absence if they discover these for
themselves.
The substantive focus of the teaching activity
is, where possible, appropriate to the content
of the module they are following.
Conceptual clarity

adds precision and rigour

facilitates construct validity

is a tool for achieving critical analysis
Construct validity





‘There’s glory for you’, said Humpty Dumpty.
‘I don’t know what you mean by “glory”,’ Alice said.
Humpty Dumpty smiled contemptuously. ‘Of course
you don’t – till I tell you. I meant “there’s a nice
knock-down argument for you”.’
‘But “glory” doesn’t mean “a nice knock-down
argument”,’ Alice objected.
‘When I use a word’, Humpty Dumpty said in a rather
scornful tone, ‘it means just what I choose it to mean
– neither more or less.’
(From: Through the looking Glass, by Lewis Carroll)
Construct validity




involves consensual acceptance and
understanding of specific terms
is threatened when researchers and research
subjects do not share the same interpretation
and understanding of key constructs
is threatened when two parties are
communicating at cross purposes
can impact upon the whole process of data
collection, data analysis, and dissemination
Developing your capacity for
pursuing construct validity
The process has four elements:
1.
identification of key concepts for
clarification
2.
conceptualisation and construct
clarification
3.
communicating and matching
constructs
4.
conceptual clarity and dissemination
Identification of key concepts
for clarification



Conceptualisation is the process whereby we
develop concepts.
It is the process of attaching meaning to
words.
It is also the product of having attached
meaning to words:

your conceptualisation of ‘milk’, for example, is the
meaning of milk that you have formulated
Identification of key concepts
for clarification
Identify what you consider to be the key
concepts within your research.
Decide which of these need clarifying:




Which concepts are potentially ambiguous?
Which concepts are susceptible to being
interpreted differently by different people?
The more nebulous the concept, the more
likely it is to need clarifying.
Concepts of tangible objects are less
susceptible to multiple interpretation than
are intangible concepts.



a study of the causes of student disaffection
Conceptualisation & construct
clarification

Examine your selected key concepts:





Decide how you interpret or define them.
What precisely do I mean by …?
It is not enough simply to ‘know’ what you mean
by something – you must be able to communicate
that meaning.
Formulate conceptual interpretations or
definitions of your key concepts.
Aim to achieve definitional precision.
Achieving definitional precision
What is a definition?
 stipulative definitions


stipulate, in precise and unambiguous words, what
you mean when you use a particular word
ostensive definitions


where a word is defined by pointing to the objects to
which it exclusively refers
e.g. in defining “education” one might point to
particular activities, or to what goes on in particular
schools
Recognising stipulative
definitions
Which of these is a stipulative definition of a chair?

A chair is a piece of furniture intended for sitting on.

A chair is a seat.

A chair has four legs, a seat and a back and is
sometimes padded. It is usually made of wood or
plastic but other materials can be used.

A chair is intended to seat people.

A chair is a piece of furniture that people may sit on.
Recognising stipulative
definitions





Stipulative definitions should explain precisely
and unambiguously what something is.
They should not simply describe something.
A stipulative definition should be exclusive –
applicable only to what it is that is being defined.
A stipulative definition should encompass
explanation of all of the features that are integral
to what is being defined.
It should incorporate consideration of the
necessary conditions for whatever is being
defined.
Stipulative definition of a chair






A chair is a piece of furniture intended for sitting on.
A chair is a seat.
A chair has four legs, a seat and a back and is
sometimes padded. It is usually made of wood or
plastic but other materials can be used.
A chair is intended to seat people.
A chair is a piece of furniture that people may sit on.
A chair is a piece of furniture with a back and is
intended to seat one person at a time.
Formulating Stipulative
Definitions

You do not necessarily need to formulate your own
definition of something.



Adopting an existing definition is perfectly
acceptable.


But you should be able to explain why you choose to adopt
it – what you like about it
Adapting an existing definition is perfectly
acceptable.


There’s no point in re-inventing the wheel.
Examine the range of definitions available in the literature.
But you should be able to explain why you choose to adapt
it – what you like and don’t like about it; why it doesn’t quite
meet your needs without being altered.
Knowing how to formulate a definition from scratch
will help you to adapt an existing one.
Formulating Stipulative
Definitions




A stipulative definition should encompass
explanation of all of the features that are
integral to what is being defined.
Start by identifying the necessary conditions
for or properties of whatever it is that you
wish to define.
First list as many conditions or properties as
you consider relevant.
Then narrow down the list to necessary
conditions or properties.
What do we mean by
‘necessary conditions’?


Necessary conditions are all those
components that constitute what the
‘thing’ is.
They are components that, if missing,
would change the nature of the ‘thing’.
Listing necessary conditions or properties:
the example of a chair

Ask yourself, for something to be called a chair, what
does it need to have?


In the case of every thing on your list ask yourself,
do all chairs have this?



Initially list as many things as spring to mind.
Cross off anything that prompts you to answer, ‘no’.
Try to think of an example of a chair that has
something that you haven’t listed.
Ask yourself, does it need to have this in order to be
called a chair?

If the answer is ‘yes’ add that thing to your list.
Properties of a chair/necessary conditions
for something to be a chair

Initial list might include:





has a seat
is a piece of furniture
has a padded seat
has four legs
is meant for sitting on
Refining the list

In the case of everything on the list ask
yourself, do all chairs have this?





has a seat
is a piece of furniture
has a padded seat
has four legs
is meant for sitting on
Refining the list

Reduce the list to include only the properties that
apply to all chairs:






has a seat
is a piece of furniture
is meant for sitting on
Try to think of an example of a chair that has
something that you haven’t listed.
Ensure that the property that you think of is common
to all chairs.
Add it to your list.
Necessary conditions or properties:
some examples

Necessary conditions for something to be a
chair







categorised as furniture
designed as a seat
intended to seat one person at a time
a back
a seat part
legs or a pedestal
Incorporate everything in the list into a
stipulative definition.
Punishment - necessary
conditions
Flew (philosopher):
Punishment must be:
 unpleasant
 for an offence or a supposed offence
 of an offender or of a supposed offender
 carried out intentionally
 carried out by those with authority to do so.
Formulate a stipulative definition of punishment.
Stipulative definition of
punishment


Punishment is an action directed at an offender or
supposed offender by one or more persons with the
authority to carry out the action, with the deliberate
intention of causing unpleasantness to the offender or
supposed offender, in response to an offence or
supposed offence committed by the targeted offender or
supposed offender.
Punishment is a deliberate action carried out by someone
in authority to carry it out on, and causing
unpleasantness to, an offender or a supposed offender
for an offence or a supposed offence.
Formulating stipulative
definitions







List the necessary conditions or properties.
Decide what category of phenomena is the concept that you
are defining
 It is a/the …
Arrange all of the necessary conditions or properties into a
statement that defines the concept, beginning with reference to
the category of phenomena.
Pay particular attention to the meaning of every single word.
Ensure that the phrases within the definition convey the precise
sense that you intend them to convey.
Test your definition by actively seeking examples of the concept
being described that are not covered by the definition.
Modify your definition until you can find no more examples of
the concept being described that are not covered by the
definition.
Achieving conceptual clarity




The key point is that you must communicate to others
what you mean when you refer to a certain term that is
susceptible to misinterpretation.
You may do this be presenting a definition or simply by
presenting your interpretation of a term.
Conceptual interpretations are as acceptable as
stipulative definitions.
Stipulative definitions are intellectually challenging to
formulate



rewarding
sense of achievement
Stipulative definitions may constitute a form of critical
analysis and may earn you extra marks …

… but only if you formulate (or adapt) them effectively.
Definitions and Conceptual
Interpretations
What’s the difference between the two?
 Stipulative definitions:




stipulate precisely what something is;
are clear and unambiguous;
relate exclusively to what is being defined.
Conceptual interpretations:




typically describe, rather than define;
may include information that supplements what would be
included in a stipulative definition;
tell us what something looks like, rather than what it is;
are what many people would categorise as ostensive
definitions.
The difference between stipulative
definitions and conceptual interpretations
Morale
 morale is: a state of mind encompassing all of the
feelings determined by the individual’s anticipation of the
extent of satisfaction of those needs which s/he perceives
as significantly affecting his/her total work situation.
 morale has levels; it ranges from high to low and these
levels may fluctuate as circumstances and situations affect
them. Morale in a work context is about how people feel
about their work. Morale is future-oriented, so it relates to
people’s feeling about their work in the future – whether
they anticipate being happy or unhappy at work…
(from: Evans, L. (2002) Reflective Practice in Educational Research, London,
Continum, p. 66)
The difference between stipulative
definitions and conceptual interpretations
Definition or interpretation?
Professional development consists of all natural learning
experiences and those conscious and planned activities
which are intended to be of direct or indirect benefit to
the individual, group or school and which contribute,
through these, to the quality of education in the
classroom. It is the process by which, alone and with
others, teachers review, renew and extend their
commitment as change agents to the moral purposes of
teaching; and by which they acquire and develop
critically the knowledge, skills, planning and practice
with children, young people and colleagues through
each phase of their teaching lives.
From: Day, C. (1999) Developing Teachers: the challenges of lifelong
learning, London, Falmer, p. 4)
Definition or Interpretation?
Conceptual interpretation:
Professional development consists of all natural
learning experiences and those conscious and planned
activities which are intended to be of direct or indirect
benefit to the individual, group or school and which
contribute, through these, to the quality of education in
the classroom.
Stipulative definition:
It is the process by which, alone and with others,
teachers review, renew and extend their commitment
as change agents to the moral purposes of teaching;
and by which they acquire and develop critically the
knowledge, skills, planning and practice with children,
young people and colleagues through each phase of
their teaching lives.
How to criticise a definition
formulated by someone else


List what you consider to be the necessary
conditions for, or properties of, the thing
being defined.
Examine the definition against this list.

Does the definition cover everything?


Does the definition incorporate reference to
conditions or properties that are not on your list?


if not, is it the definition or your list that’s deficient ?
if so, is it the definition or your list that’s deficient ?
Formulate a critique.

Explain why you consider the definition to be
deficient.
Activity




Formulate a stipulative definition of a concept
that is key to your work.
Don’t tell anyone what concept you are
defining.
In small groups, each person is to read out
her/his definition, but without the name of
the concept being defined.
The rest of the group must work out what
concept is being defined.
An alternative activity

Prepare two sets of cards:
1.
2.

with the name of a concept on the card
with the stipulative definition of the
concept on the card – but not the name
of the concept
Students to work in groups, sorting
the cards into pairs, matching the
concepts with their definitions.
Categorisation and coding




The process of grouping, sorting and classifying: of formulating
and identifying categories and placing phenomena into these
categories.
Codes categorise information at different levels.
Categorisation is probably the most difficult research/scholarship
skill
An invaluable analytical tool



Useful for essay/dissertation writing:


allows you to analyse your own findings
allows you to criticise other people’s analyses and categorisations
subheadings
The difficulty lies in formulating categories that:


don’t overlap
are on the same coding ‘level’ (as each other).
From: Evans, L. (2002) Reflective Practice in
Educational Research: developing advanced skills,
London, Continuum: (Chapter 8: Tools of the Trade:
developing coding and categorisation skills)
What makes effective categorisation so
difficult are the two requirements that
categories be both exhaustive and exclusive.
Gillham (2000a, p. 60) explains: ‘(A)
requirement for the derivation of categories is
that they should be exhaustive. But another
requirement is that they should be exclusive,
i.e. that the kind of statements that go into
one category clearly belong there and
couldn’t really go anywhere else’.
An example from a student’s work
characteristics of effective school leadership:









well-organised
trustworthiness and honesty
approachable
positive attitudes
interpersonal skills
strong leadership qualities
abilities to make decisions
innovative
knowledge of the staff and
children









knowledge of teaching and
learning
knowledge of the school
vision
supportive
high expectations
encouragement of team
work
a leading professional
lead by setting an example
development of the staff
From: Evans, L. (2002) Reflective Practice in
Educational Research: developing advanced skills,
London, Continuum: (Chapter 8: Tools of the Trade:
developing coding and categorisation skills)
This list is flawed in several ways that I
identify below. As it stands it will not
contribute towards the development of theory
without further treatment. To move on to a
deeper level of analysis the student who
compiled it will need to refine and reduce it
by sorting into categories. To develop into an
analytical researcher – an ‘extended’
professional – you need to move beyond the
level of simply presenting research findings as
lists compiled from raw data. You need to
delve deeper.
Categorisation skills




recognition and elimination of overlap;
recognition and incorporation of outlier
and atypical cases;
application of appropriate levels of the
basis of categorisation,
presentation of categories.
Recognition and elimination of
overlap

The number of categories you have is an
important clue:




too many categories implies a flawed
categorisation;
categories are likely to overlap.
Use a systematic approach to identifying and
eliminating overlap.
I suggest two complementary approaches:


systematic comparative pairing
putting things into drawers
Systematic comparative
pairing


Involves systematically comparing each of your categories
with each other, in turn, to examine whether or not they
overlap
In the case of each pairing the comparison is a two-way
process:



ask the same question twice in relation to each pairing:

or


category A is compared with B
category B is compared with A
is this category a sub-category of the other? (A of B, then B of A)
is this category subsumed within the other? (A of B, then B of A)
If the answer to both is, ‘No’, then ask:

could this category be considered to be essentially the same thing
as the other?
Putting things into drawers

Involves the pictorial representation of your
categorisation as a chest of drawers





Examine your units of data.
‘Put away’ the data into whichever drawers they belong
in
The purpose is to ascertain if specific units of data may
feasibly be placed into more than one drawer.
If so, the categorisation is flawed.


one drawer for each category generated – labelled appropriately
It involves overlap.
Effective categorisation will be represented by drawers in
only one of which any specific portion or unit of data – at
whatever level of analysis you are working – belongs.
Putting things into drawers
spoons
forks
teaspoons
knives
cutlery
cutting
implements
Issues that matter to education
professionals, in their work contexts
Figure 2: Sorting data into categories of issues that matter to people: an
example of pictorial representation of categorisation as labelled drawers.
7
1
4
3
2
5
6
8